00010000:00010000:1.0F:1769076291.376033:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0F:1769076292.399852:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076293.424878:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0F:1769076293.743904:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076293.743927:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076293.743932:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076293.743942:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1976 00000100:00080000:0.0:1769076293.743971:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076293.743978:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076293.744016:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961351552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076293.744066:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076293.744070:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076293.744072:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076293.744078:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1976 00000100:00080000:0.0:1769076293.744084:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076293.744090:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076293.744103:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076293.744121:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961351680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076293.744135:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076293.744140:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076293.744143:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076293.744148:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1976 00000100:00080000:0.0:1769076293.744155:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076293.744162:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076293.744170:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076293.744191:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961351808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076293.744193:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961351552:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769076293.744295:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076293.744365:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076293/real 1769076293] req@ffff895e4383d500 x1855008961351552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076293.744392:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961351552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076293.744407:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076293.744414:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076293.744420:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961351552:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076293.744431:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961351680:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076293.744465:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076293.744494:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076293/real 1769076293] req@ffff895e4383f800 x1855008961351680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076293.744515:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961351680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076293.744527:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076293.744532:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076293.744537:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961351680:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076293.744573:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961351808:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076293.744603:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076293.744620:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076293/real 1769076293] req@ffff895e4383e300 x1855008961351808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076293.744844:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961351808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076293.744859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076293.744865:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076293.744873:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961351808:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769076294.447994:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769076295.471908:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1769076296.495986:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1769076296.652926:0:28730:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-sec test 25a: test save and reload nodemap config ========================================================== 05:04:55 (1769076295) 00010000:00010000:1.0:1769076297.520057:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769076298.544001:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076298.863849:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076298.863869:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076298.863876:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076298.863894:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1981 00000100:00080000:0.0:1769076298.863913:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076298.863926:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076298.863981:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961351936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076298.864026:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076298.864034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076298.864038:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076298.864046:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1981 00000100:00080000:0.0:1769076298.864055:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076298.864063:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076298.864081:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076298.864124:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961352064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076298.864126:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961351936:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076298.864161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076298.864167:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076298.864170:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076298.864177:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1981 00000100:00080000:0.0:1769076298.864184:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076298.864212:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076298.864222:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076298.864258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076298.864272:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961352192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076298.864344:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076298/real 1769076298] req@ffff895e4383e300 x1855008961351936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076298.864384:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961351936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076298.864406:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076298.864416:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076298.864427:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961351936:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076298.864472:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961352064:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076298.864524:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076298.864555:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076298/real 1769076298] req@ffff895e4383f800 x1855008961352064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076298.864615:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961352064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076298.864633:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076298.864641:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076298.864650:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961352064:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076298.864667:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961352192:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076298.864714:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076298.864863:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076298/real 1769076298] req@ffff895e4383d500 x1855008961352192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076298.864893:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961352192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076298.864911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076298.864918:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076298.864925:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961352192:192.168.204.117@tcp:38:kworker.0 00000080:00200008:1.0:1769076299.371704:0:28878:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff895e4ad1e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1769076299.371718:0:28878:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff895e4ad1e3c8),name=/ 00000002:00010000:1.0:1769076299.371793:0:28878:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1769076299.371862:0:28878:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff895e42f9a800 lock: ffff895e47563e00/0xb4d3f8fc6477c768 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 28878 initiator: MDT0 00010000:00010000:1.0:1769076299.371875:0:28878:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff895e42f9a800 lock: ffff895e47563e00/0xb4d3f8fc6477c768 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 28878 initiator: MDT0 00010000:00010000:1.0:1769076299.371887:0:28878:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff895e42f9a800 lock: ffff895e47563e00/0xb4d3f8fc6477c768 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 28878 initiator: MDT0 00000100:00080000:1.0:1769076299.371916:0:28878:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff895e4713e680 x1855008961352320/t0(0) o101->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'lsof.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1769076299.371930:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076299.568000:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076300.399917:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076300.591998:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076301.423970:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0F:1769076301.615945:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076302.447938:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076302.640077:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076303.471924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076303.663975:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076303.983861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076303.983874:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076303.983879:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076303.983892:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1986 00000100:00080000:0.0:1769076303.983906:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076303.983916:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076303.983961:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961352448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076303.984003:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076303.984008:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076303.984011:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076303.984016:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1986 00000100:00080000:0.0:1769076303.984020:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076303.984024:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076303.984037:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076303.984052:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961352576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076303.984067:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076303.984093:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769076303.984093:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961352448:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076303.984098:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076303.984115:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1986 00000100:00080000:0.0:1769076303.984120:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076303.984126:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076303.984133:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076303.984174:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961352704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076303.984216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076303.984265:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076303/real 1769076303] req@ffff895e4383ea00 x1855008961352448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076303.984288:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961352448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076303.984314:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076303.984320:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076303.984326:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961352448:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076303.984336:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961352576:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076303.984367:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076303.984382:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076303/real 1769076303] req@ffff895e4383ca80 x1855008961352576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076303.984398:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961352576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076303.984408:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076303.984413:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076303.984417:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961352576:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076303.984447:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961352704:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076303.984466:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076303.984477:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076303/real 1769076303] req@ffff895e4383dc00 x1855008961352704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076303.984491:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961352704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076303.984501:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076303.984504:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076303.984508:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961352704:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076304.495963:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076304.688021:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076305.519941:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076305.712005:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076306.543897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076306.735963:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076307.567903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076307.760021:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076308.591886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076308.783949:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076309.103965:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076309.103978:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076309.103984:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076309.103999:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1991 00000100:00080000:2.0:1769076309.104014:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076309.104112:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076309.104160:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961352832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076309.104200:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076309.104205:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076309.104207:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076309.104212:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1991 00000100:00080000:2.0:1769076309.104217:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076309.104223:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076309.104235:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076309.104250:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961352960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076309.104267:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076309.104273:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769076309.104274:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961352832:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076309.104289:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076309.104309:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1991 00000100:00080000:2.0:1769076309.104313:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076309.104318:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076309.104327:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076309.104372:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961353088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076309.104438:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076309.104503:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076309/real 1769076309] req@ffff895e44e29180 x1855008961352832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076309.104531:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961352832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076309.104547:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076309.104554:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076309.104560:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961352832:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076309.104600:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961352960:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076309.104628:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076309.104644:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076309/real 1769076309] req@ffff895e44e29500 x1855008961352960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076309.104659:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961352960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076309.104671:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076309.104675:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076309.104679:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961352960:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076309.104710:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961353088:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076309.104737:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076309.104801:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076309/real 1769076309] req@ffff895e44e28a80 x1855008961353088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076309.104818:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961353088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076309.104829:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076309.104833:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076309.104838:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961353088:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076309.615866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076309.807932:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076310.639828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076310.831962:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076311.663958:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076311.855996:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076312.687907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076312.880024:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076313.711864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076313.904049:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076314.223933:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076314.223946:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076314.223951:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076314.223963:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1996 00000100:00080000:0.0:1769076314.224019:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076314.224033:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076314.224092:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961353216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076314.224144:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076314.224150:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076314.224153:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076314.224158:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1996 00000100:00080000:0.0:1769076314.224164:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076314.224170:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076314.224203:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076314.224215:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961353216:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076314.224226:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961353344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076314.224266:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076314.224272:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076314.224277:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076314.224285:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1996 00000100:00080000:0.0:1769076314.224294:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076314.224303:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076314.224314:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076314.224316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076314.224349:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961353472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076314.224380:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076314/real 1769076314] req@ffff895e4383ce00 x1855008961353216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076314.224422:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961353216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076314.224443:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076314.224452:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076314.224461:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961353216:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076314.224498:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961353344:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076314.224550:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076314.224583:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076314/real 1769076314] req@ffff895e4383c000 x1855008961353344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076314.224602:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961353344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076314.224614:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076314.224619:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076314.224623:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961353344:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076314.224632:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961353472:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076314.224658:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076314.224679:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076314/real 1769076314] req@ffff895e4383f480 x1855008961353472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076314.224703:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961353472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076314.224721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076314.224728:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076314.224736:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961353472:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076314.735874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076314.928040:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076315.759882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076315.951995:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076316.783872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076316.976013:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076317.807909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076318.000001:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076318.831864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076319.024036:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076319.343993:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076319.344011:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076319.344019:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076319.344035:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2001 00000100:00080000:3.0:1769076319.344050:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076319.344079:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076319.344122:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961353600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076319.344164:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076319.344169:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076319.344171:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076319.344176:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2001 00000100:00080000:3.0:1769076319.344182:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076319.344186:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076319.344200:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076319.344224:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961353728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076319.344242:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961353600:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076319.344252:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076319.344262:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076319.344266:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076319.344277:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2001 00000100:00080000:3.0:1769076319.344286:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076319.344295:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076319.344306:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076319.344328:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961353856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076319.344346:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076319.344412:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076319/real 1769076319] req@ffff895e453d6680 x1855008961353600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076319.344443:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961353600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076319.344458:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076319.344464:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076319.344470:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961353600:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076319.344519:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961353728:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076319.344553:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076319.344570:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076319/real 1769076319] req@ffff895e453d6300 x1855008961353728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076319.344586:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961353728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076319.344597:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076319.344601:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076319.344606:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961353728:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076319.344627:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961353856:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076319.344654:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076319.344666:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076319/real 1769076319] req@ffff895e453d4380 x1855008961353856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076319.344681:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961353856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076319.344691:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076319.344695:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076319.344698:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961353856:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076319.855860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076320.048064:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076320.879911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076321.072058:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076321.903828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076322.095919:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076322.927989:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076323.120036:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076323.951940:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076324.144024:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076324.463939:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076324.463951:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076324.463957:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076324.463968:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2007 00000100:00080000:0.0:1769076324.463982:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076324.463990:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076324.464048:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961353984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076324.464085:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076324.464109:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076324.464111:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076324.464118:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2007 00000100:00080000:0.0:1769076324.464128:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076324.464137:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076324.464154:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076324.464163:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961353984:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076324.464185:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961354112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076324.464212:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076324.464219:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076324.464223:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076324.464231:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2007 00000100:00080000:0.0:1769076324.464239:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076324.464245:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076324.464253:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076324.464261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076324.464277:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961354240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076324.464325:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076324/real 1769076324] req@ffff895e4383c380 x1855008961353984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076324.464348:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961353984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076324.464362:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076324.464368:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076324.464374:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961353984:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076324.464402:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961354112:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076324.464427:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076324.464440:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076324/real 1769076324] req@ffff895e4383f100 x1855008961354112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076324.464468:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961354112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076324.464479:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076324.464483:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076324.464486:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961354112:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076324.464493:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961354240:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076324.464516:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076324.464527:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076324/real 1769076324] req@ffff895e4383fb80 x1855008961354240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076324.464540:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961354240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076324.464549:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076324.464552:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076324.464555:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961354240:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076324.975886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076325.167995:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076325.999918:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076326.191924:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076327.023952:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076327.216015:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076328.047916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076328.239971:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076329.071909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076329.264002:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076329.583937:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076329.583954:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076329.583961:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076329.583974:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2012 00000100:00080000:0.0:1769076329.583992:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076329.584003:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076329.584096:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961354368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076329.584139:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076329.584143:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076329.584146:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076329.584151:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2012 00000100:00080000:0.0:1769076329.584157:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076329.584163:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076329.584177:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076329.584194:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961354496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076329.584208:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076329.584219:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769076329.584219:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961354368:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076329.584230:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076329.584242:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2012 00000100:00080000:0.0:1769076329.584246:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076329.584253:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076329.584260:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076329.584280:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961354624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076329.584316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076329.584360:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076329/real 1769076329] req@ffff895e4383e680 x1855008961354368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076329.584376:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961354368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076329.584403:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076329.584407:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076329.584410:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961354368:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076329.584433:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961354496:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076329.584453:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076329.584462:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076329/real 1769076329] req@ffff895e4383d880 x1855008961354496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076329.584471:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961354496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076329.584477:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076329.584480:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076329.584482:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961354496:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076329.584487:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961354624:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076329.584498:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076329.584504:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076329/real 1769076329] req@ffff895e4383c700 x1855008961354624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076329.584512:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961354624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076329.584526:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076329.584528:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076329.584531:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961354624:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076330.095900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076330.287950:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076331.119917:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076331.312043:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076332.143857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076332.336053:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076333.167850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076333.360039:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076334.191927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076334.383994:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076334.703922:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076334.703948:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076334.703953:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076334.703967:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2017 00000100:00080000:3.0:1769076334.703981:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076334.703990:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076334.704037:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961354752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076334.704083:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076334.704088:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076334.704091:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076334.704096:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2017 00000100:00080000:3.0:1769076334.704100:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076334.704105:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076334.704116:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076334.704140:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961354880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076334.704155:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076334.704159:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076334.704187:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076334.704194:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2017 00000100:00080000:3.0:1769076334.704200:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076334.704209:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961354752:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076334.704212:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076334.704224:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076334.704242:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961355008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076334.704360:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961354880:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076334.704401:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961355008:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076335.215925:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076335.408037:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076336.239868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076336.431904:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076337.263923:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076337.455926:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769076337.776074:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769076337.776128:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076337.776150:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769076337.776194:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769076337.776204:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:2.0:1769076337.776267:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076334/real 1769076337] req@ffff895e453d6300 x1855008961355008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076337.776316:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961355008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076337.776338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076337.776348:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076337.776358:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961355008:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076337.776373:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076334/real 1769076337] req@ffff895e453d6d80 x1855008961354752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076337.776400:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961354752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076337.776429:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076337.776436:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076337.776443:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961354752:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076337.776457:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076334/real 1769076337] req@ffff895e453d4380 x1855008961354880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076337.776480:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961354880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076337.776515:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076337.776523:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076337.776530:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961354880:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076338.287900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076338.479981:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076339.311909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076339.503969:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076339.823874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076339.823889:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076339.823895:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076339.823908:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2022 00000100:00080000:3.0:1769076339.823923:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076339.823932:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076339.823984:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961355136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076339.824035:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076339.824043:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076339.824046:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076339.824054:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2022 00000100:00080000:3.0:1769076339.824062:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076339.824070:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076339.824086:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076339.824156:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961355136:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076339.824163:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961355264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076339.824196:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076339.824203:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076339.824208:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076339.824217:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2022 00000100:00080000:3.0:1769076339.824251:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1769076339.824260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769076339.824270:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076339.824285:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076339.824318:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961355392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076339.824339:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076339/real 1769076339] req@ffff895e453d6680 x1855008961355136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076339.824382:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961355136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076339.824405:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076339.824414:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076339.824422:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961355136:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076339.824463:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961355264:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076339.824511:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076339.824537:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076339/real 1769076339] req@ffff895e453d5180 x1855008961355264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076339.824604:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961355264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076339.824622:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076339.824630:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076339.824637:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961355264:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076339.824667:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961355392:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076339.824712:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076339.824736:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076339/real 1769076339] req@ffff895e453d5880 x1855008961355392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076339.824822:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961355392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076339.824837:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076339.824843:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076339.824849:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961355392:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076340.335880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076340.528048:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076341.359912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076341.552033:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076342.383863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076342.576005:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076343.408020:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076343.600024:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076344.431989:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076344.624039:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076344.943990:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076344.944008:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076344.944015:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076344.944031:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2027 00000100:00080000:0.0:1769076344.944049:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076344.944061:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076344.944126:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961355520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076344.944175:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076344.944180:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076344.944182:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076344.944188:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2027 00000100:00080000:0.0:1769076344.944192:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076344.944197:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076344.944210:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076344.944228:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961355648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076344.944245:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076344.944275:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961355520:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076344.944282:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076344.944294:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076344.944308:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2027 00000100:00080000:0.0:1769076344.944319:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076344.944331:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076344.944343:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076344.944373:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961355776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076344.944393:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076344.944483:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076344/real 1769076344] req@ffff895e4383df80 x1855008961355520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076344.944512:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961355520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076344.944527:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076344.944533:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076344.944539:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961355520:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076344.944550:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961355648:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076344.944579:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076344.944595:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076344/real 1769076344] req@ffff895e4383ed80 x1855008961355648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076344.944641:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961355648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076344.944659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076344.944666:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076344.944672:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961355648:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076344.944718:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961355776:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076344.944824:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076344.944844:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076344/real 1769076344] req@ffff895e4383d180 x1855008961355776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076344.944861:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961355776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076344.944871:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076344.944876:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076344.944880:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961355776:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076345.455954:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076345.648067:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076346.479876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076346.671908:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076347.503884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076347.695942:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076348.527902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076348.720017:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076349.551873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076349.743997:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076350.063879:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076350.063894:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076350.063901:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076350.063914:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2032 00000100:00080000:3.0:1769076350.063930:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076350.063940:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076350.063989:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961355904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076350.064030:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076350.064037:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076350.064040:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076350.064047:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2032 00000100:00080000:3.0:1769076350.064066:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076350.064088:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076350.064099:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961355904:192.168.204.117@tcp:250:kworker.0 00000008:00100000:3.0:1769076350.064102:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076350.064126:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961356032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076350.064140:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076350.064144:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076350.064146:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076350.064150:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2032 00000100:00080000:3.0:1769076350.064156:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076350.064160:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076350.064165:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076350.064177:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961356160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076350.064199:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076350.064262:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076350/real 1769076350] req@ffff895e453d4700 x1855008961355904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076350.064291:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961355904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076350.064330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076350.064337:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076350.064344:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961355904:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076350.064400:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961356032:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076350.064452:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076350.064470:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076350/real 1769076350] req@ffff895e453d7100 x1855008961356032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076350.064485:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961356032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076350.064514:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076350.064518:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076350.064523:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961356032:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076350.064531:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961356160:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076350.064554:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076350.064565:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076350/real 1769076350] req@ffff895e453d7b80 x1855008961356160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076350.064579:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961356160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076350.064589:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076350.064592:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076350.064596:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961356160:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076350.575919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076350.767983:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076351.599903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076351.792080:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076352.623851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076352.815869:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076353.647921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076353.839983:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076354.671944:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076354.864020:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076355.183941:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076355.183966:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076355.183975:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076355.183995:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2037 00000100:00080000:1.0:1769076355.184017:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076355.184032:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076355.184111:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961356288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076355.184170:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076355.184211:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076355.184216:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076355.184227:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2037 00000100:00080000:1.0:1769076355.184238:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076355.184254:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076355.184256:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961356288:192.168.204.117@tcp:250:kworker.0 00000008:00100000:1.0:1769076355.184277:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076355.184348:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961356416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076355.184376:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076355.184383:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076355.184386:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076355.184395:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2037 00000100:00080000:1.0:1769076355.184405:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076355.184413:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:2.0:1769076355.184423:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:1.0:1769076355.184432:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076355.184466:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961356544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076355.184504:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076355/real 1769076355] req@ffff895e4713ed80 x1855008961356288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076355.184536:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961356288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076355.184559:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076355.184568:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076355.184576:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961356288:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076355.184630:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961356416:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076355.184677:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076355.184696:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076355/real 1769076355] req@ffff895e4713d880 x1855008961356416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076355.184714:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961356416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076355.184726:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076355.184731:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076355.184736:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961356416:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076355.184791:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961356544:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076355.184821:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076355.184836:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076355/real 1769076355] req@ffff895e4713e300 x1855008961356544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076355.184867:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961356544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076355.184895:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076355.184900:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076355.184904:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961356544:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076355.695882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076355.887971:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076356.719842:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076356.911948:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076357.743945:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076357.936016:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076358.768064:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076358.960039:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076359.791933:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076359.984099:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076360.303888:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076360.303899:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076360.303904:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076360.303913:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2042 00000100:00080000:3.0:1769076360.303925:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076360.303933:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076360.303971:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961356672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076360.304005:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076360.304009:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076360.304028:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076360.304033:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2042 00000100:00080000:3.0:1769076360.304037:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076360.304042:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076360.304053:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076360.304073:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961356800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076360.304074:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961356672:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076360.304095:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076360.304098:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076360.304100:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076360.304106:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2042 00000100:00080000:3.0:1769076360.304110:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076360.304113:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076360.304117:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076360.304130:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961356928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076360.304157:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076360.304231:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076360/real 1769076360] req@ffff895e453d5500 x1855008961356672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076360.304253:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961356672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076360.304264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076360.304269:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076360.304273:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961356672:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076360.304298:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961356800:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076360.304322:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076360.304335:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076360/real 1769076360] req@ffff895e453d7480 x1855008961356800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076360.304347:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961356800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076360.304372:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076360.304376:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076360.304380:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961356800:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076360.304386:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961356928:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076360.304405:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076360.304415:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076360/real 1769076360] req@ffff895e453d4e00 x1855008961356928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076360.304426:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961356928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076360.304433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076360.304435:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076360.304438:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961356928:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076360.815901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076361.008090:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076361.839906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076362.031999:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076362.863861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076363.056060:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076363.887901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076364.079947:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076364.911888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076365.103983:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076365.423819:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076365.423829:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076365.423833:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076365.423843:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2047 00000100:00080000:3.0:1769076365.423853:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076365.423879:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076365.423940:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961357056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076365.424017:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076365.424020:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076365.424022:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076365.424026:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2047 00000100:00080000:3.0:1769076365.424029:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076365.424033:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076365.424041:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076365.424053:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961357184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076365.424061:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076365.424064:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076365.424067:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076365.424070:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2047 00000100:00080000:3.0:1769076365.424072:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076365.424076:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076365.424080:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076365.424088:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961357312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076365.424113:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961357056:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769076365.424236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076365.424285:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076365/real 1769076365] req@ffff895e453d7800 x1855008961357056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076365.424308:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961357056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076365.424322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076365.424328:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076365.424344:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961357056:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076365.424355:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961357184:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076365.424384:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076365.424413:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076365/real 1769076365] req@ffff895e453d5f80 x1855008961357184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076365.424424:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961357184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076365.424431:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076365.424434:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076365.424437:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961357184:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076365.424460:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961357312:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076365.424474:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076365.424481:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076365/real 1769076365] req@ffff895e453d4a80 x1855008961357312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076365.424490:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961357312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076365.424497:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076365.424499:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076365.424502:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961357312:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076365.936201:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076366.127904:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076366.959836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076367.151944:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076367.983876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076368.176054:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076369.007858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076369.200037:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076370.031910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076370.223979:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076370.543876:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076370.543893:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076370.543901:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076370.543916:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2053 00000100:00080000:3.0:1769076370.543953:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076370.543962:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076370.544004:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961357440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076370.544064:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076370.544070:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076370.544073:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076370.544078:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2053 00000100:00080000:3.0:1769076370.544083:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076370.544088:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076370.544100:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076370.544120:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961357568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076370.544135:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076370.544140:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076370.544144:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076370.544152:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2053 00000100:00100000:2.0:1769076370.544162:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961357440:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076370.544164:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076370.544185:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076370.544198:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076370.544229:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961357696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076370.544262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076370.544328:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076370/real 1769076370] req@ffff895e453d5c00 x1855008961357440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076370.544352:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961357440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076370.544366:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076370.544372:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076370.544377:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961357440:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076370.544388:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961357568:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076370.544415:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076370.544429:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076370/real 1769076370] req@ffff895e453d4a80 x1855008961357568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076370.544444:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961357568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076370.544455:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076370.544458:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076370.544462:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961357568:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076370.544493:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961357696:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076370.544517:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076370.544537:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076370/real 1769076370] req@ffff895e453d5f80 x1855008961357696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076370.544560:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961357696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076370.544605:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076370.544612:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076370.544620:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961357696:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076371.055910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076371.248014:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076372.079946:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076372.272093:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076373.103900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076373.295939:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076374.127990:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076374.320114:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076375.151997:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076375.344029:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076375.663930:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076375.663964:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076375.663970:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076375.663983:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2058 00000100:00080000:1.0:1769076375.663997:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076375.664006:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076375.664053:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961357824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076375.664091:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076375.664096:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076375.664117:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076375.664122:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2058 00000100:00080000:1.0:1769076375.664128:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076375.664133:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076375.664146:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076375.664162:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961357952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076375.664178:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076375.664182:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076375.664185:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076375.664193:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2058 00000100:00080000:1.0:1769076375.664200:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076375.664208:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076375.664218:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076375.664223:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961357824:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076375.664245:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961358080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076375.664313:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076375.664362:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076375/real 1769076375] req@ffff895e4713f100 x1855008961357824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076375.664386:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961357824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076375.664399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076375.664405:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076375.664410:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961357824:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076375.664419:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961357952:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076375.664443:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076375.664457:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076375/real 1769076375] req@ffff895e4713ea00 x1855008961357952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076375.664472:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961357952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076375.664482:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076375.664486:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076375.664490:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961357952:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076375.664536:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961358080:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076375.664562:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076375.664575:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076375/real 1769076375] req@ffff895e4713d500 x1855008961358080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076375.664590:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961358080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076375.664600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076375.664604:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076375.664608:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961358080:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076376.175853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076376.368001:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076377.199882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076377.391984:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076378.223874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076378.415977:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076379.247868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076379.440022:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076380.271946:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076380.464041:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076380.783932:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076380.783970:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076380.783975:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076380.783987:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2063 00000100:00080000:0.0:1769076380.784002:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076380.784011:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076380.784055:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961358208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076380.784103:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076380.784107:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076380.784110:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076380.784115:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2063 00000100:00080000:0.0:1769076380.784119:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076380.784123:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076380.784135:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076380.784150:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961358336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076380.784163:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076380.784168:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076380.784170:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076380.784175:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2063 00000100:00080000:0.0:1769076380.784179:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076380.784184:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076380.784190:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076380.784200:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961358464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076380.784217:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961358208:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769076380.784309:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076380.784358:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076380/real 1769076380] req@ffff895e4383d180 x1855008961358208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076380.784382:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961358208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076380.784395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076380.784401:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076380.784406:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961358208:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076380.784415:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961358336:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076380.784439:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076380.784452:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076380/real 1769076380] req@ffff895e4383ed80 x1855008961358336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076380.784484:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961358336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076380.784495:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076380.784499:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076380.784503:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961358336:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076380.784536:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961358464:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076380.784561:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076380.784574:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076380/real 1769076380] req@ffff895e4383df80 x1855008961358464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076380.784589:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961358464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076380.784599:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076380.784602:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076380.784606:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961358464:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076381.295972:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076381.487999:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076382.319893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076382.511958:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076383.343946:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076383.535963:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076384.367977:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076384.560014:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076385.391932:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076385.583974:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076385.903951:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076385.903964:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076385.903970:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076385.903983:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2068 00000100:00080000:1.0:1769076385.903997:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076385.904007:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076385.904064:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961358592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076385.904104:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076385.904109:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076385.904111:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076385.904116:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2068 00000100:00080000:1.0:1769076385.904121:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076385.904125:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076385.904137:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076385.904171:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961358720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076385.904192:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076385.904196:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076385.904198:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076385.904201:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2068 00000100:00080000:1.0:1769076385.904204:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076385.904207:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076385.904213:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076385.904217:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961358592:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076385.904251:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961358848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076385.904296:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076385.904338:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076385/real 1769076385] req@ffff895e4713ce00 x1855008961358592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076385.904378:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961358592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076385.904442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076385.904448:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076385.904453:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961358592:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076385.904492:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961358720:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076385.904550:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076385.904571:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076385/real 1769076385] req@ffff895e4713f480 x1855008961358720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076385.904610:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961358720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076385.904621:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076385.904625:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076385.904629:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961358720:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076385.904637:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961358848:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076385.904671:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076385.904684:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076385/real 1769076385] req@ffff895e4713ca80 x1855008961358848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076385.904697:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961358848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076385.904706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076385.904709:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076385.904713:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961358848:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076386.415924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076386.607995:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076387.439952:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076387.631989:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076388.463939:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076388.655947:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076389.487947:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076389.679980:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076390.511863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076390.703960:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076391.023899:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076391.023935:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076391.023941:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076391.023953:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2073 00000100:00080000:3.0:1769076391.023966:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076391.023976:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076391.024018:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961358976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076391.024056:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076391.024061:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076391.024064:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076391.024069:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2073 00000100:00080000:3.0:1769076391.024073:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076391.024077:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076391.024089:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076391.024104:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961359104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076391.024117:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076391.024121:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076391.024123:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076391.024127:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2073 00000100:00080000:3.0:1769076391.024131:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076391.024136:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076391.024142:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076391.024155:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961359232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076391.024189:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961358976:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769076391.024289:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076391.024357:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076391/real 1769076391] req@ffff895e453d7800 x1855008961358976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076391.024393:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961358976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076391.024414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076391.024422:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076391.024431:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961358976:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076391.024443:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961359104:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076391.024478:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076391.024496:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076391/real 1769076391] req@ffff895e453d4e00 x1855008961359104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076391.024511:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961359104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076391.024522:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076391.024526:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076391.024531:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961359104:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076391.024561:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961359232:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076391.024581:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076391.024593:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076391/real 1769076391] req@ffff895e453d7480 x1855008961359232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076391.024622:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961359232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076391.024632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076391.024636:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076391.024640:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961359232:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076391.535866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076391.727983:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076392.559902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076392.751948:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076393.583875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076393.775969:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076394.607947:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076394.799967:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076395.631996:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076395.823968:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076396.143826:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076396.143836:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076396.143841:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076396.143850:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2078 00000100:00080000:2.0:1769076396.143860:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076396.143866:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076396.143897:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961359360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076396.143921:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076396.143925:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076396.143927:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076396.143931:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2078 00000100:00080000:2.0:1769076396.143934:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076396.143939:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076396.143948:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076396.143957:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961359360:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076396.143962:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961359488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076396.143993:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076396.143996:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076396.143998:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076396.144002:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2078 00000100:00080000:2.0:1769076396.144013:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1769076396.144017:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:2.0:1769076396.144018:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076396.144022:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076396.144033:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961359616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076396.144059:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076396/real 1769076396] req@ffff895e44e29c00 x1855008961359360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076396.144079:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961359360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076396.144090:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076396.144094:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076396.144097:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961359360:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076396.144118:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961359488:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076396.144139:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076396.144149:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076396/real 1769076396] req@ffff895e44e2b800 x1855008961359488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076396.144157:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961359488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076396.144163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076396.144165:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076396.144167:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961359488:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076396.144171:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961359616:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076396.144180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076396.144194:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076396/real 1769076396] req@ffff895e44e2bb80 x1855008961359616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076396.144202:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961359616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076396.144208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076396.144211:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076396.144214:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961359616:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076396.655979:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076396.848029:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076397.680014:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076397.872029:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076398.703928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076398.896032:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076399.727947:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076399.919971:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076400.751909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076400.943976:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076401.263861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076401.263876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076401.263882:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076401.263896:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2083 00000100:00080000:2.0:1769076401.263911:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076401.263922:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076401.263966:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961359744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076401.263997:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076401.264001:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076401.264017:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076401.264023:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2083 00000100:00080000:2.0:1769076401.264029:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076401.264037:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076401.264054:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076401.264055:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961359744:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076401.264085:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961359872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076401.264108:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076401.264114:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076401.264117:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076401.264123:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2083 00000100:00080000:2.0:1769076401.264139:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076401.264145:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076401.264152:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076401.264171:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961360000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076401.264195:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961359872:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076401.264234:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961360000:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076401.775957:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076401.967984:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076402.799850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076402.992025:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076403.823810:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076404.015883:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769076404.335989:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:0.0:1769076404.336026:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769076404.336041:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:0.0:1769076404.336082:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:0.0:1769076404.336093:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769076404.336134:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076401/real 1769076404] req@ffff895e44e29c00 x1855008961360000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076404.336156:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961360000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076404.336165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076404.336169:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076404.336172:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961360000:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076404.336184:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076401/real 1769076404] req@ffff895e44e2b800 x1855008961359872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076404.336196:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961359872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076404.336221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076404.336227:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076404.336232:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961359872:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076404.336238:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076401/real 1769076404] req@ffff895e44e2bb80 x1855008961359744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076404.336252:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961359744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076404.336259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076404.336262:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076404.336264:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961359744:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076404.847817:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076405.039995:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076405.871871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076406.064003:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076406.383885:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076406.383900:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076406.383906:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076406.383920:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2088 00000100:00080000:2.0:1769076406.383935:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076406.383946:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076406.383990:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961360128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076406.384054:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076406.384058:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076406.384061:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076406.384066:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2088 00000100:00080000:2.0:1769076406.384072:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076406.384076:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076406.384087:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076406.384104:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961360256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076406.384116:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076406.384121:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076406.384123:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076406.384127:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2088 00000100:00080000:2.0:1769076406.384130:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076406.384134:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076406.384139:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076406.384149:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961360384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076406.384209:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961360128:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769076406.384317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076406.384377:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076406/real 1769076406] req@ffff895e44e28a80 x1855008961360128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076406.384413:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961360128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076406.384464:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076406.384473:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076406.384481:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961360128:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076406.384498:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961360256:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076406.384568:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076406.384595:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076406/real 1769076406] req@ffff895e44e29500 x1855008961360256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076406.384622:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961360256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076406.384639:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076406.384646:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076406.384653:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961360256:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076406.384668:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961360384:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076406.384709:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076406.384732:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076406/real 1769076406] req@ffff895e44e29180 x1855008961360384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076406.384808:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961360384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076406.384826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076406.384833:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076406.384839:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961360384:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076406.895940:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076407.088050:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076407.919925:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076408.111942:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076408.943945:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076409.135998:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076409.967921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076410.160030:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076410.991863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076411.183990:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076411.503873:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076411.503887:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076411.503893:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076411.503929:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2094 00000100:00080000:2.0:1769076411.503945:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076411.503956:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076411.504027:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961360512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076411.504070:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076411.504075:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076411.504078:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076411.504083:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2094 00000100:00080000:2.0:1769076411.504088:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076411.504093:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076411.504106:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076411.504125:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961360640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076411.504141:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076411.504147:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961360512:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076411.504148:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076411.504155:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076411.504165:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2094 00000100:00080000:2.0:1769076411.504169:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076411.504173:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076411.504181:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076411.504205:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961360768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076411.504251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076411.504315:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076411/real 1769076411] req@ffff895e44e28e00 x1855008961360512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076411.504393:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961360512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076411.504406:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076411.504412:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076411.504417:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961360512:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076411.504448:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961360640:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076411.504481:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076411.504497:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076411/real 1769076411] req@ffff895e44e2b480 x1855008961360640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076411.504512:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961360640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076411.504522:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076411.504526:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076411.504530:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961360640:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076411.504537:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961360768:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076411.504556:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076411.504566:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076411/real 1769076411] req@ffff895e44e29f80 x1855008961360768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076411.504579:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961360768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076411.504631:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076411.504639:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076411.504646:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961360768:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076412.015832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076412.207994:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076413.039889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076413.231968:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076414.063873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076414.256005:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076415.087895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076415.280002:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076416.112003:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076416.303993:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076416.623882:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076416.623896:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076416.623902:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076416.623914:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2099 00000100:00080000:3.0:1769076416.623929:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076416.623939:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076416.623994:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961360896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076416.624052:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076416.624058:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076416.624061:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076416.624067:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2099 00000100:00080000:3.0:1769076416.624073:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076416.624079:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076416.624093:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076416.624114:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961361024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076416.624130:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076416.624134:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769076416.624145:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961360896:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076416.624158:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076416.624167:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2099 00000100:00080000:3.0:1769076416.624173:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076416.624178:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076416.624186:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076416.624205:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961361152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076416.624309:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076416.624364:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076416/real 1769076416] req@ffff895e453d5500 x1855008961360896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076416.624400:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961360896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076416.624415:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076416.624421:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076416.624426:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961360896:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076416.624458:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961361024:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076416.624491:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076416.624507:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076416/real 1769076416] req@ffff895e453d7b80 x1855008961361024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076416.624523:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961361024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076416.624534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076416.624538:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076416.624542:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961361024:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076416.624561:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961361152:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076416.624606:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076416.624621:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076416/real 1769076416] req@ffff895e453d7100 x1855008961361152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076416.624636:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961361152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076416.624647:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076416.624651:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076416.624655:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961361152:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076417.135909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076417.328026:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076418.159905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076418.352039:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076419.183908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076419.376012:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076420.207848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076420.400020:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076421.231888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076421.424066:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076421.743834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076421.743847:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076421.743851:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076421.743863:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2104 00000100:00080000:3.0:1769076421.743877:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076421.743885:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076421.743955:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961361280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076421.744007:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076421.744015:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076421.744019:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076421.744027:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2104 00000100:00080000:3.0:1769076421.744035:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076421.744043:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076421.744068:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076421.744089:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961361408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076421.744133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076421.744141:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769076421.744144:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961361280:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076421.744145:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076421.744161:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2104 00000100:00080000:3.0:1769076421.744199:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076421.744209:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076421.744218:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076421.744240:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961361536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076421.744284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076421.744353:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076421/real 1769076421] req@ffff895e453d4700 x1855008961361280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076421.744389:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961361280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076421.744409:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076421.744417:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076421.744426:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961361280:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076421.744463:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961361408:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076421.744512:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076421.744531:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076421/real 1769076421] req@ffff895e453d5880 x1855008961361408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076421.744547:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961361408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076421.744558:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076421.744562:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076421.744567:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961361408:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076421.744591:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961361536:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076421.744612:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076421.744622:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076421/real 1769076421] req@ffff895e453d5180 x1855008961361536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076421.744635:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961361536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076421.744644:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076421.744647:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076421.744650:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961361536:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076422.255917:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076422.447990:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076423.279897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076423.472030:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076424.303937:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076424.496004:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076425.327896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076425.519956:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076426.351874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076426.543971:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076426.863905:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076426.863939:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076426.863945:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076426.863957:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2109 00000100:00080000:2.0:1769076426.863983:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076426.863993:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076426.864035:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961361664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076426.864073:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076426.864079:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076426.864105:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076426.864113:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2109 00000100:00080000:2.0:1769076426.864122:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076426.864130:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076426.864146:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076426.864157:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961361664:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076426.864170:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961361792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076426.864190:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076426.864195:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076426.864199:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076426.864205:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2109 00000100:00080000:2.0:1769076426.864230:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076426.864238:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:3.0:1769076426.864244:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:2.0:1769076426.864247:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076426.864266:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961361920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076426.864300:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076426/real 1769076426] req@ffff895e44e2b100 x1855008961361664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076426.864332:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961361664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076426.864351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076426.864359:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076426.864366:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961361664:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076426.864403:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961361792:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076426.864453:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076426.864471:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076426/real 1769076426] req@ffff895e44e2a300 x1855008961361792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076426.864500:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961361792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076426.864511:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076426.864515:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076426.864520:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961361792:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076426.864528:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961361920:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076426.864556:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076426.864569:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076426/real 1769076426] req@ffff895e44e2aa00 x1855008961361920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076426.864583:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961361920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076426.864593:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076426.864597:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076426.864601:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961361920:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076427.375928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076427.568120:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076428.399980:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076428.592008:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076429.423891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076429.616091:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076430.447856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076430.639999:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076431.471875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076431.664033:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076431.983867:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076431.983881:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076431.983887:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076431.983899:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2114 00000100:00080000:2.0:1769076431.983913:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076431.983922:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076431.983968:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961362048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076431.984046:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076431.984052:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076431.984055:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076431.984061:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2114 00000100:00080000:2.0:1769076431.984067:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076431.984074:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076431.984087:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076431.984108:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961362176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076431.984126:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961362048:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076431.984129:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076431.984145:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076431.984150:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076431.984157:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2114 00000100:00080000:2.0:1769076431.984161:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076431.984165:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076431.984193:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076431.984212:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961362304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076431.984241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076431.984294:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076431/real 1769076431] req@ffff895e44e29880 x1855008961362048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076431.984335:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961362048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076431.984349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076431.984355:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076431.984361:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961362048:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076431.984392:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961362176:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076431.984422:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076431.984437:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076431/real 1769076431] req@ffff895e44e28000 x1855008961362176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076431.984452:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961362176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076431.984463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076431.984467:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076431.984472:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961362176:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076431.984479:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961362304:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076431.984496:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076431.984507:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076431/real 1769076431] req@ffff895e44e28380 x1855008961362304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076431.984521:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961362304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076431.984548:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076431.984552:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076431.984557:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961362304:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076432.495871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076432.688044:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076433.519915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076433.711998:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076434.543845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076434.736034:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076435.567920:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076435.760022:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076436.591893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076436.784002:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076437.103879:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076437.103893:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076437.103899:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076437.103934:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2119 00000100:00080000:2.0:1769076437.103952:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076437.103965:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076437.104020:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961362432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076437.104070:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076437.104077:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076437.104081:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076437.104090:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2119 00000100:00080000:2.0:1769076437.104098:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076437.104108:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076437.104125:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076437.104146:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961362432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076437.104159:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961362560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076437.104180:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076437.104185:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076437.104188:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076437.104193:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2119 00000100:00080000:2.0:1769076437.104225:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076437.104232:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:3.0:1769076437.104241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:2.0:1769076437.104245:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076437.104269:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961362688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076437.104308:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076437/real 1769076437] req@ffff895e44e28380 x1855008961362432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076437.104334:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961362432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076437.104348:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076437.104353:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076437.104359:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961362432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076437.104390:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961362560:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076437.104457:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076437.104482:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076437/real 1769076437] req@ffff895e44e28000 x1855008961362560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076437.104508:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961362560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076437.104542:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076437.104547:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076437.104552:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961362560:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076437.104561:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961362688:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076437.104593:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076437.104616:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076437/real 1769076437] req@ffff895e44e29880 x1855008961362688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076437.104641:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961362688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076437.104657:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076437.104665:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076437.104672:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961362688:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076437.615875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076437.807993:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076438.640047:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076438.831998:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076439.663848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076439.856017:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076440.687963:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076440.879987:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076441.711857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076441.903971:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076442.223874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076442.223886:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076442.223891:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076442.223902:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2124 00000100:00080000:3.0:1769076442.223914:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076442.223922:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076442.223964:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961362816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076442.223996:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076442.224029:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076442.224032:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076442.224037:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2124 00000100:00080000:3.0:1769076442.224042:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076442.224048:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076442.224056:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961362816:192.168.204.117@tcp:250:kworker.0 00000008:00100000:3.0:1769076442.224062:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076442.224081:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961362944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076442.224095:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076442.224098:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076442.224101:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076442.224105:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2124 00000100:00080000:3.0:1769076442.224109:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076442.224114:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076442.224118:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076442.224129:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961363072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076442.224153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076442.224228:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076442/real 1769076442] req@ffff895e453d6680 x1855008961362816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076442.224250:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961362816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076442.224262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076442.224267:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076442.224272:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961362816:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076442.224338:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961362944:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076442.224400:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076442.224421:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076442/real 1769076442] req@ffff895e453d4380 x1855008961362944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076442.224443:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961362944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076442.224469:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076442.224474:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076442.224494:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961362944:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076442.224520:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961363072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076442.224557:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076442.224572:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076442/real 1769076442] req@ffff895e453d6d80 x1855008961363072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076442.224585:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961363072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076442.224594:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076442.224597:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076442.224601:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961363072:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076442.735855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076442.927881:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076443.759896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076443.952024:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076444.783853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076444.976049:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076445.807884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076446.000034:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076446.831841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076447.023905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076447.343923:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076447.343937:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076447.343943:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076447.343955:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2129 00000100:00080000:1.0:1769076447.343990:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076447.343998:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076447.344054:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961363200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076447.344078:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076447.344080:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076447.344082:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076447.344084:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2129 00000100:00080000:1.0:1769076447.344087:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076447.344107:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076447.344117:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076447.344129:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961363200:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076447.344134:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961363328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076447.344149:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076447.344152:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076447.344154:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076447.344156:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2129 00000100:00080000:1.0:1769076447.344160:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076447.344170:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076447.344175:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076447.344184:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961363456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076447.344197:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076447.344231:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076447/real 1769076447] req@ffff895e4713f800 x1855008961363200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076447.344245:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961363200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076447.344253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076447.344256:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076447.344259:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961363200:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076447.344279:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961363328:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076447.344304:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076447.344312:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076447/real 1769076447] req@ffff895e4713dc00 x1855008961363328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076447.344321:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961363328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076447.344327:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076447.344329:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076447.344331:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961363328:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076447.344335:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961363456:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076447.344345:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076447.344350:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076447/real 1769076447] req@ffff895e4713d180 x1855008961363456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076447.344358:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961363456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076447.344363:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076447.344365:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076447.344366:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961363456:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076447.855949:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076448.047911:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076448.879820:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076449.071960:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076449.903920:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076450.095902:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076450.927814:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076451.119993:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076451.951975:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076452.144003:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076452.463953:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076452.463969:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076452.463976:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076452.464017:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2135 00000100:00080000:3.0:1769076452.464034:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076452.464044:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076452.464109:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961363584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076452.464149:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076452.464153:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076452.464156:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076452.464160:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2135 00000100:00080000:3.0:1769076452.464164:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076452.464167:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076452.464179:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076452.464195:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961363712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076452.464207:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076452.464211:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076452.464213:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076452.464228:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2135 00000100:00080000:3.0:1769076452.464231:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076452.464235:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076452.464240:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076452.464245:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961363584:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076452.464254:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961363840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076452.464375:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076452.464436:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076452/real 1769076452] req@ffff895e453d6300 x1855008961363584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076452.464478:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961363584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076452.464511:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076452.464517:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076452.464521:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961363584:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076452.464531:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961363712:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076452.464557:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076452.464570:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076452/real 1769076452] req@ffff895e453d6d80 x1855008961363712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076452.464583:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961363712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076452.464592:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076452.464595:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076452.464599:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961363712:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076452.464632:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961363840:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076452.464649:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076452.464705:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076452/real 1769076452] req@ffff895e453d4380 x1855008961363840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076452.464718:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961363840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076452.464741:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076452.464800:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076452.464805:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961363840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076452.975846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076453.167951:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076453.999840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076454.192048:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076455.023895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076455.215964:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076456.047902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076456.240012:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076457.071926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076457.263946:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076457.583871:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076457.583885:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076457.583891:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076457.583904:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2140 00000100:00080000:3.0:1769076457.583918:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076457.583929:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076457.583989:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961363968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076457.584038:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076457.584046:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076457.584050:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076457.584057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2140 00000100:00080000:3.0:1769076457.584067:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076457.584077:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076457.584094:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076457.584108:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961363968:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076457.584124:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961364096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076457.584150:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076457.584155:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076457.584158:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076457.584164:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2140 00000100:00080000:3.0:1769076457.584188:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076457.584195:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:2.0:1769076457.584201:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:3.0:1769076457.584229:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076457.584261:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961364224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076457.584263:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076457/real 1769076457] req@ffff895e453d6680 x1855008961363968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076457.584328:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961363968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076457.584344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076457.584352:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076457.584358:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961363968:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076457.584388:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961364096:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076457.584415:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076457.584430:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076457/real 1769076457] req@ffff895e453d5180 x1855008961364096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076457.584445:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961364096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076457.584456:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076457.584472:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076457.584477:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961364096:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076457.584486:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961364224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076457.584512:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076457.584525:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076457/real 1769076457] req@ffff895e453d5880 x1855008961364224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076457.584539:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961364224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076457.584549:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076457.584552:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076457.584556:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961364224:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076458.095912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076458.288003:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076459.119900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076459.311972:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076460.143874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076460.335962:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076461.167919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076461.359942:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076462.191903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076462.383986:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076462.703902:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076462.703916:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076462.703921:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076462.703933:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2145 00000100:00080000:0.0:1769076462.703970:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076462.703980:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076462.704027:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961364352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076462.704067:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076462.704072:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076462.704074:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076462.704079:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2145 00000100:00080000:0.0:1769076462.704084:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076462.704088:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076462.704100:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076462.704116:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961364480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076462.704131:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076462.704136:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076462.704138:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769076462.704146:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961364352:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076462.704148:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2145 00000100:00080000:0.0:1769076462.704159:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076462.704183:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076462.704189:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076462.704201:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961364608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076462.704264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076462.704321:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076462/real 1769076462] req@ffff895e4383c700 x1855008961364352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076462.704466:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961364352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076462.704531:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076462.704537:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076462.704554:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961364352:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076462.704583:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961364480:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076462.704616:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076462.704633:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076462/real 1769076462] req@ffff895e4383d880 x1855008961364480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076462.704648:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961364480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076462.704658:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076462.704661:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076462.704665:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961364480:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076462.704672:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961364608:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076462.704690:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076462.704700:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076462/real 1769076462] req@ffff895e4383e680 x1855008961364608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076462.704712:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961364608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076462.704721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076462.704724:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076462.704727:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961364608:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076463.215906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076463.408024:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076464.239876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076464.431902:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076465.263879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076465.456025:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076466.287811:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076466.479868:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076467.311897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076467.503945:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076467.823882:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076467.823917:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076467.823922:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076467.823934:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2150 00000100:00080000:0.0:1769076467.823948:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076467.823957:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076467.824006:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961364736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076467.824059:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076467.824067:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076467.824072:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076467.824080:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2150 00000100:00080000:0.0:1769076467.824087:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076467.824095:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076467.824111:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076467.824135:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961364864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076467.824146:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961364736:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076467.824178:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076467.824184:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076467.824187:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076467.824194:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2150 00000100:00080000:0.0:1769076467.824201:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076467.824208:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076467.824214:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076467.824234:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961364992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076467.824313:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961364864:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076467.824356:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961364992:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076468.335857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076468.528023:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076469.359877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076469.552044:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076470.383881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076470.575962:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769076470.896166:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769076470.896240:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076470.896267:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769076470.896317:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769076470.896329:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:2.0:1769076470.896467:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076467/real 1769076470] req@ffff895e4383f100 x1855008961364864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076470.896504:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961364864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076470.896520:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076470.896527:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076470.896532:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961364864:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076470.896542:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076467/real 1769076470] req@ffff895e4383c380 x1855008961364992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076470.896556:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961364992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076470.896565:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076470.896569:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076470.896618:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961364992:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076470.896628:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076467/real 1769076470] req@ffff895e4383fb80 x1855008961364736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076470.896648:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961364736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076470.896663:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076470.896671:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076470.896678:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961364736:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076471.407932:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076471.600044:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076472.431889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076472.623957:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076472.943865:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076472.943874:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076472.943878:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076472.943886:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2155 00000100:00080000:3.0:1769076472.943895:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076472.943901:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076472.943928:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961365120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076472.943952:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076472.943955:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076472.943956:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076472.943959:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2155 00000100:00080000:3.0:1769076472.943961:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076472.943963:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076472.943971:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076472.943983:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961365248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076472.943995:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076472.944004:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961365120:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076472.944016:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076472.944020:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076472.944025:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2155 00000100:00080000:3.0:1769076472.944030:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076472.944035:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076472.944041:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076472.944054:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961365376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076472.944076:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076472.944116:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076472/real 1769076472] req@ffff895e453d4700 x1855008961365120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076472.944145:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961365120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076472.944156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076472.944161:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076472.944166:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961365120:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076472.944192:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961365248:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076472.944227:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076472.944250:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076472/real 1769076472] req@ffff895e453d7100 x1855008961365248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076472.944264:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961365248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076472.944273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076472.944276:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076472.944280:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961365248:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076472.944288:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961365376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076472.944343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076472.944356:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076472/real 1769076472] req@ffff895e453d7b80 x1855008961365376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076472.944369:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961365376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076472.944378:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076472.944381:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076472.944385:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961365376:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076473.455907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076473.647978:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076474.479967:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076474.672029:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076475.504003:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076475.695994:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076476.527930:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076476.719958:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076477.552019:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076477.743982:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076478.063954:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076478.063970:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076478.063977:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076478.063992:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2160 00000100:00080000:0.0:1769076478.064008:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076478.064019:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076478.064070:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961365504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076478.064131:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076478.064136:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076478.064138:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076478.064143:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2160 00000100:00080000:0.0:1769076478.064149:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076478.064154:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076478.064166:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076478.064184:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961365632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076478.064198:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076478.064203:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076478.064205:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076478.064212:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2160 00000100:00080000:0.0:1769076478.064218:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076478.064289:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076478.064301:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076478.064342:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961365504:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076478.064343:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961365760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076478.064513:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076478.064600:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076478/real 1769076478] req@ffff895e4383f480 x1855008961365504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076478.064627:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961365504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076478.064643:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076478.064649:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076478.064655:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961365504:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076478.064666:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961365632:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076478.064692:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076478.064706:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076478/real 1769076478] req@ffff895e4383c000 x1855008961365632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076478.064720:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961365632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076478.064731:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076478.064735:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076478.064739:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961365632:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076478.064845:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961365760:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076478.064869:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076478.064901:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076478/real 1769076478] req@ffff895e4383ce00 x1855008961365760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076478.064955:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961365760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076478.064966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076478.064971:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076478.064975:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961365760:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076478.576025:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076478.768029:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076479.599854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076479.792000:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076480.623996:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076480.815951:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076481.647918:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076481.839937:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076482.671910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076482.863989:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076483.183984:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076483.183998:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076483.184005:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076483.184017:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2165 00000100:00080000:3.0:1769076483.184032:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076483.184041:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076483.184087:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961365888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076483.184127:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076483.184132:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076483.184135:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076483.184163:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2165 00000100:00080000:3.0:1769076483.184168:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076483.184172:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076483.184184:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076483.184200:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961366016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076483.184235:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076483.184240:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076483.184243:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076483.184249:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2165 00000100:00080000:3.0:1769076483.184255:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1769076483.184267:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961365888:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076483.184267:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076483.184284:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076483.184303:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961366144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076483.184370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076483.184418:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076483/real 1769076483] req@ffff895e453d5500 x1855008961365888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076483.184442:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961365888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076483.184456:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076483.184462:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076483.184467:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961365888:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076483.184477:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961366016:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076483.184501:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076483.184515:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076483/real 1769076483] req@ffff895e453d7480 x1855008961366016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076483.184553:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961366016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076483.184565:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076483.184569:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076483.184573:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961366016:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076483.184605:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961366144:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076483.184630:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076483.184643:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076483/real 1769076483] req@ffff895e453d4e00 x1855008961366144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076483.184657:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961366144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076483.184667:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076483.184670:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076483.184674:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961366144:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076483.695896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076483.888032:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076484.719909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076484.912013:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076485.743890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076485.935998:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076486.767851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076486.960127:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076487.791851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076487.984106:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076488.303872:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076488.303898:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076488.303904:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076488.303917:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2170 00000100:00080000:3.0:1769076488.303931:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076488.303940:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076488.303986:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961366272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076488.304024:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076488.304028:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076488.304031:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076488.304036:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2170 00000100:00080000:3.0:1769076488.304040:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076488.304044:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076488.304108:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076488.304129:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961366400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076488.304147:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076488.304156:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1769076488.304157:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961366272:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076488.304159:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076488.304175:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2170 00000100:00080000:3.0:1769076488.304196:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076488.304202:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076488.304209:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076488.304226:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961366528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076488.304266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076488.304316:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076488/real 1769076488] req@ffff895e453d7800 x1855008961366272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076488.304363:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961366272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076488.304377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076488.304383:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076488.304389:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961366272:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076488.304421:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961366400:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076488.304453:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076488.304469:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076488/real 1769076488] req@ffff895e453d5f80 x1855008961366400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076488.304484:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961366400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076488.304495:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076488.304500:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076488.304504:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961366400:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076488.304511:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961366528:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076488.304529:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076488.304540:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076488/real 1769076488] req@ffff895e453d4a80 x1855008961366528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076488.304554:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961366528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076488.304580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076488.304585:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076488.304589:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961366528:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076488.815937:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076489.007990:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076489.839899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076490.031953:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076490.863825:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076491.055888:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076491.887857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076492.079913:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076492.911866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076493.103881:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076493.423855:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076493.423865:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076493.423868:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076493.423876:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2175 00000100:00080000:3.0:1769076493.423886:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076493.423892:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076493.423925:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961366656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076493.423956:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076493.423958:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076493.423960:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076493.423963:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2175 00000100:00080000:3.0:1769076493.423995:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076493.424000:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076493.424010:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076493.424013:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961366656:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076493.424029:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961366784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076493.424043:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076493.424047:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076493.424050:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076493.424054:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2175 00000100:00080000:3.0:1769076493.424059:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076493.424063:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076493.424068:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1769076493.424084:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076493.424092:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961366912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769076493.424121:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076493/real 1769076493] req@ffff895e453d5c00 x1855008961366656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076493.424137:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961366656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076493.424144:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076493.424147:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076493.424150:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961366656:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076493.424169:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961366784:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076493.424183:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076493.424191:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076493/real 1769076493] req@ffff895e453d4a80 x1855008961366784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076493.424207:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961366784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076493.424216:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076493.424219:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076493.424223:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961366784:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076493.424240:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961366912:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076493.424276:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076493.424286:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076493/real 1769076493] req@ffff895e453d5f80 x1855008961366912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076493.424295:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961366912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076493.424300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076493.424302:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076493.424304:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961366912:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076493.935886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076494.128024:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076494.959823:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076495.151889:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076495.983816:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076496.175946:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076497.007848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076497.199871:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076498.031822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076498.223880:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076498.543853:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076498.543862:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076498.543865:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076498.543873:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2181 00000100:00080000:1.0:1769076498.543911:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076498.543919:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076498.543955:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961367040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076498.543985:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076498.543988:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076498.543990:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076498.543993:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2181 00000100:00080000:1.0:1769076498.543996:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076498.543999:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076498.544006:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076498.544017:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961367168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076498.544024:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076498.544027:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076498.544028:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076498.544031:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2181 00000100:00080000:1.0:1769076498.544033:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076498.544035:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076498.544045:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961367040:192.168.204.117@tcp:250:kworker.0 00000008:00100000:1.0:1769076498.544057:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076498.544070:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961367296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076498.544112:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076498.544143:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076498/real 1769076498] req@ffff895e4713c380 x1855008961367040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076498.544156:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961367040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076498.544164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076498.544167:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076498.544170:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961367040:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076498.544175:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961367168:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076498.544188:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076498.544195:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076498/real 1769076498] req@ffff895e4713df80 x1855008961367168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076498.544203:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961367168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076498.544208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076498.544210:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076498.544212:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961367168:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076498.544232:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961367296:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076498.544242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076498.544248:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076498/real 1769076498] req@ffff895e4713d180 x1855008961367296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076498.544281:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961367296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076498.544287:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076498.544289:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076498.544291:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961367296:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076499.055907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076499.247972:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076500.079850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076500.271954:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076501.103816:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076501.295899:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076502.127866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076502.319942:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076503.151826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076503.343903:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076503.663836:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076503.663844:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076503.663848:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076503.663855:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2186 00000100:00080000:2.0:1769076503.663872:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076503.663878:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076503.663910:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961367424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076503.663941:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076503.663945:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076503.663963:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076503.663968:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2186 00000100:00080000:2.0:1769076503.663972:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076503.663977:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961367424:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076503.663978:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076503.663989:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076503.664003:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961367552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076503.664012:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076503.664014:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076503.664016:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076503.664019:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2186 00000100:00080000:2.0:1769076503.664031:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076503.664033:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076503.664037:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076503.664042:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076503.664048:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961367680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076503.664078:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076503/real 1769076503] req@ffff895e44e2aa00 x1855008961367424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076503.664093:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961367424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076503.664100:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076503.664112:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076503.664116:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961367424:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076503.664143:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961367552:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076503.664162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076503.664170:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076503/real 1769076503] req@ffff895e44e2a300 x1855008961367552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076503.664179:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961367552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076503.664185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076503.664187:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076503.664190:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961367552:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076503.664194:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961367680:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076503.664204:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076503.664210:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076503/real 1769076503] req@ffff895e44e2b100 x1855008961367680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076503.664217:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961367680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076503.664222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076503.664224:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076503.664226:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961367680:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076504.175854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076504.367900:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076505.199879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076505.391919:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076506.223858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076506.415926:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076507.247839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076507.439896:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076508.271908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076508.463905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076508.783848:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076508.783891:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076508.783896:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076508.783907:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2191 00000100:00080000:2.0:1769076508.783919:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076508.783928:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076508.783969:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961367808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076508.783997:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076508.784000:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076508.784002:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076508.784005:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2191 00000100:00080000:2.0:1769076508.784008:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076508.784011:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076508.784020:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076508.784032:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961367936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076508.784041:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076508.784044:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076508.784045:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076508.784050:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2191 00000100:00080000:2.0:1769076508.784057:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076508.784063:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076508.784063:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961367808:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769076508.784076:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076508.784090:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961368064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076508.784139:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076508.784176:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076508/real 1769076508] req@ffff895e44e29f80 x1855008961367808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076508.784204:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961367808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076508.784216:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076508.784221:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076508.784225:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961367808:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076508.784255:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961367936:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076508.784280:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076508.784291:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076508/real 1769076508] req@ffff895e44e2b480 x1855008961367936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076508.784299:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961367936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076508.784308:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076508.784311:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076508.784315:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961367936:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076508.784321:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961368064:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076508.784342:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076508.784354:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076508/real 1769076508] req@ffff895e44e28e00 x1855008961368064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076508.784367:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961368064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076508.784386:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076508.784390:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076508.784393:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961368064:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076509.295930:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076509.487952:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076510.320008:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076510.511903:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076511.343891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076511.535912:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076512.367823:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076512.559879:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076513.391869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076513.583957:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076513.903842:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076513.903854:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076513.903866:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076513.903878:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2196 00000100:00080000:1.0:1769076513.903888:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076513.903897:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076513.903941:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961368192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076513.903970:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076513.903974:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076513.903976:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076513.903981:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2196 00000100:00080000:1.0:1769076513.903984:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076513.903989:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076513.904000:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076513.904019:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961368320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076513.904019:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961368192:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769076513.904037:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076513.904055:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076513.904058:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076513.904063:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2196 00000100:00080000:1.0:1769076513.904068:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076513.904073:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076513.904079:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076513.904090:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769076513.904095:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961368448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076513.904133:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076513/real 1769076513] req@ffff895e4713dc00 x1855008961368192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076513.904153:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961368192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076513.904164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076513.904169:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076513.904174:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961368192:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076513.904198:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961368320:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076513.904235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076513.904249:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076513/real 1769076513] req@ffff895e4713f800 x1855008961368320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076513.904263:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961368320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076513.904280:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076513.904284:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076513.904288:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961368320:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076513.904296:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961368448:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076513.904320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076513.904332:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076513/real 1769076513] req@ffff895e4713ca80 x1855008961368448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076513.904344:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961368448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076513.904353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076513.904357:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076513.904360:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961368448:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076514.415885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076514.607910:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076515.439860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076515.631926:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076516.463835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076516.655901:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076517.487847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076517.679860:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076518.511810:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076518.704006:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076519.023953:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076519.023970:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076519.023977:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076519.023993:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2201 00000100:00080000:2.0:1769076519.024011:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076519.024022:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076519.024096:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961368576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076519.024147:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076519.024152:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076519.024154:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076519.024183:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2201 00000100:00080000:2.0:1769076519.024188:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076519.024193:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076519.024207:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076519.024224:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961368704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076519.024238:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076519.024269:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961368576:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076519.024271:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076519.024281:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076519.024309:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2201 00000100:00080000:2.0:1769076519.024316:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076519.024322:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076519.024330:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076519.024355:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961368832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076519.024407:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076519.024462:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076519/real 1769076519] req@ffff895e44e29180 x1855008961368576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076519.024485:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961368576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076519.024517:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076519.024523:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076519.024529:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961368576:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076519.024580:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961368704:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076519.024609:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076519.024622:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076519/real 1769076519] req@ffff895e44e29500 x1855008961368704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076519.024635:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961368704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076519.024643:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076519.024647:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076519.024650:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961368704:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076519.024677:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961368832:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076519.024693:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076519.024702:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076519/real 1769076519] req@ffff895e44e28a80 x1855008961368832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076519.024713:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961368832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076519.024720:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076519.024723:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076519.024726:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961368832:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076519.535950:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076519.728068:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076520.559860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076520.751889:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076521.583960:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076521.776074:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076522.607890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076522.799973:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076523.631898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076523.824031:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076524.144029:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076524.144052:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076524.144061:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076524.144093:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2206 00000100:00080000:3.0:1769076524.144114:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076524.144125:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076524.144186:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961368960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076524.144235:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076524.144240:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076524.144243:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076524.144248:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2206 00000100:00080000:3.0:1769076524.144254:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076524.144261:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076524.144276:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076524.144293:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961369088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076524.144309:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769076524.144314:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961368960:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076524.144315:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076524.144361:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076524.144369:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2206 00000100:00080000:3.0:1769076524.144376:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076524.144382:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076524.144392:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076524.144423:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961369216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769076524.144484:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076524.144555:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076524/real 1769076524] req@ffff895e453d7800 x1855008961368960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076524.144582:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961368960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076524.144598:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076524.144604:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076524.144610:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961368960:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076524.144647:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961369088:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076524.144687:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076524.144720:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076524/real 1769076524] req@ffff895e453d4e00 x1855008961369088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076524.144737:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961369088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076524.144782:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076524.144787:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076524.144792:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961369088:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076524.144801:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961369216:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769076524.144837:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076524.144853:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076524/real 1769076524] req@ffff895e453d7480 x1855008961369216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076524.144868:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961369216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769076524.144878:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769076524.144882:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769076524.144886:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961369216:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076524.655855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076524.848050:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076525.679971:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076525.871943:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076526.703880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076526.896015:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076527.727857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076527.919934:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076528.751845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076528.943887:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076529.263828:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076529.263837:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076529.263856:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076529.263864:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2211 00000100:00080000:2.0:1769076529.263874:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076529.263881:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076529.263910:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961369344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076529.263933:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076529.263936:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076529.263937:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076529.263940:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2211 00000100:00080000:2.0:1769076529.263943:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076529.263947:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076529.263955:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076529.263966:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961369344:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076529.263967:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961369472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076529.263981:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076529.263984:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076529.263985:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076529.263988:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2211 00000100:00080000:2.0:1769076529.263991:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076529.263994:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076529.263997:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076529.264007:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961369600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076529.264031:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076529.264078:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076529/real 1769076529] req@ffff895e44e2bb80 x1855008961369344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076529.264099:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961369344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076529.264111:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076529.264114:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076529.264117:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961369344:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076529.264158:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961369472:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076529.264190:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076529.264206:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076529/real 1769076529] req@ffff895e44e2b800 x1855008961369472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076529.264240:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961369472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076529.264270:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076529.264273:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076529.264277:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961369472:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076529.264283:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961369600:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076529.264305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076529.264316:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076529/real 1769076529] req@ffff895e44e29c00 x1855008961369600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076529.264325:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961369600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076529.264331:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076529.264333:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076529.264335:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961369600:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076529.775928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076529.968040:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076530.799875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076530.991966:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076531.823886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076532.015988:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076532.847878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076533.039921:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076533.871822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076534.064051:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076534.383908:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076534.383916:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076534.383920:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076534.383927:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2216 00000100:00080000:0.0:1769076534.383937:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076534.383942:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076534.383977:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961369728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076534.384009:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076534.384013:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076534.384015:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076534.384019:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2216 00000100:00080000:0.0:1769076534.384023:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076534.384027:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076534.384041:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076534.384052:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961369856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076534.384068:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961369728:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076534.384074:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076534.384078:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076534.384080:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076534.384083:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2216 00000100:00080000:0.0:1769076534.384087:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076534.384090:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076534.384094:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076534.384102:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961369984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076534.384223:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961369856:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076534.384333:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961369984:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076534.895832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076535.087975:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076535.919818:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076536.111919:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076536.943865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076537.135946:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769076537.455984:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769076537.456030:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769076537.456046:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769076537.456075:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769076537.456080:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769076537.456140:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076534/real 1769076537] req@ffff895e4383ca80 x1855008961369856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076537.456166:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961369856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076537.456179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076537.456184:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076537.456189:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961369856:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076537.456198:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076534/real 1769076537] req@ffff895e4383ea00 x1855008961369984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076537.456210:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961369984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076537.456218:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076537.456222:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076537.456239:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961369984:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076537.456248:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076534/real 1769076537] req@ffff895e4383dc00 x1855008961369728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076537.456261:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961369728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076537.456286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076537.456290:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076537.456294:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961369728:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076537.967821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076538.159884:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076538.991854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076539.183987:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076539.503914:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076539.503923:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076539.503927:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076539.503936:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2222 00000100:00080000:2.0:1769076539.503945:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076539.503951:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076539.503982:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961370112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076539.504010:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076539.504014:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076539.504017:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076539.504022:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2222 00000100:00080000:2.0:1769076539.504042:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076539.504048:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076539.504058:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076539.504073:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961370240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076539.504081:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961370112:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076539.504089:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076539.504093:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076539.504094:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076539.504098:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2222 00000100:00080000:2.0:1769076539.504101:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076539.504104:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076539.504108:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076539.504131:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961370368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076539.504169:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076539.504249:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076539/real 1769076539] req@ffff895e44e29c00 x1855008961370112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076539.504271:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961370112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076539.504281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076539.504286:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076539.504290:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961370112:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076539.504320:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961370240:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076539.504347:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076539.504361:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076539/real 1769076539] req@ffff895e44e2b800 x1855008961370240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076539.504374:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961370240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076539.504383:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076539.504395:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076539.504399:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961370240:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076539.504406:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961370368:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076539.504430:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076539.504443:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076539/real 1769076539] req@ffff895e44e2bb80 x1855008961370368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076539.504456:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961370368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076539.504463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076539.504467:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076539.504470:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961370368:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076540.015822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076540.207837:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076541.039848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076541.231861:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076542.063815:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076542.255879:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076543.087842:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076543.279863:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076544.111830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076544.303868:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076544.623838:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076544.623851:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076544.623871:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076544.623882:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2227 00000100:00080000:1.0:1769076544.623892:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076544.623899:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076544.623947:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961370496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076544.623970:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076544.623974:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076544.623975:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076544.623978:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2227 00000100:00080000:1.0:1769076544.623981:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076544.623984:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076544.623992:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076544.624015:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961370624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076544.624040:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076544.624042:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076544.624043:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076544.624046:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2227 00000100:00080000:1.0:1769076544.624049:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076544.624052:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076544.624055:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076544.624060:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961370496:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076544.624065:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961370752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076544.624167:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076544.624202:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076544/real 1769076544] req@ffff895e4713f480 x1855008961370496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076544.624216:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961370496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076544.624227:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076544.624232:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076544.624237:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961370496:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076544.624255:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961370624:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076544.624282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076544.624295:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076544/real 1769076544] req@ffff895e4713ce00 x1855008961370624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076544.624305:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961370624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076544.624311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076544.624314:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076544.624316:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961370624:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076544.624341:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961370752:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076544.624364:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076544.624377:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076544/real 1769076544] req@ffff895e4713d500 x1855008961370752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076544.624390:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961370752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076544.624416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076544.624419:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076544.624423:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961370752:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076545.135866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076545.327865:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076546.159882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076546.351966:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076547.183850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076547.375939:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076548.207851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076548.399959:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076549.231831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076549.423966:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076549.743952:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076549.743960:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076549.743964:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076549.743971:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2232 00000100:00080000:0.0:1769076549.743981:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076549.743986:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076549.744018:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961370880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076549.744067:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076549.744070:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076549.744085:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076549.744089:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2232 00000100:00080000:0.0:1769076549.744092:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076549.744097:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076549.744106:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076549.744116:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961371008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076549.744126:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076549.744132:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961370880:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076549.744133:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076549.744139:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076549.744153:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2232 00000100:00080000:0.0:1769076549.744155:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076549.744158:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076549.744161:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076549.744171:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961371136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076549.744230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076549.744277:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076549/real 1769076549] req@ffff895e4383d500 x1855008961370880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076549.744307:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961370880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076549.744317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076549.744321:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076549.744325:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961370880:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076549.744351:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961371008:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076549.744369:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076549.744378:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076549/real 1769076549] req@ffff895e4383f800 x1855008961371008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076549.744387:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961371008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076549.744393:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076549.744396:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076549.744398:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961371008:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076549.744402:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961371136:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076549.744412:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076549.744418:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076549/real 1769076549] req@ffff895e4383e300 x1855008961371136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076549.744425:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961371136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076549.744440:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076549.744443:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076549.744445:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961371136:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076550.255870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076550.447948:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076551.279826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076551.471890:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076552.303916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076552.495939:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076553.327870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076553.519921:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076554.351914:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076554.543937:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076554.863840:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076554.863849:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076554.863853:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076554.863861:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2237 00000100:00080000:2.0:1769076554.863870:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076554.863875:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076554.863905:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961371264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076554.863931:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076554.863944:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076554.863946:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076554.863950:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2237 00000100:00080000:2.0:1769076554.863954:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076554.863960:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076554.863970:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076554.863980:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961371264:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076554.863987:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961371392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076554.864000:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076554.864002:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076554.864004:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076554.864007:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2237 00000100:00080000:2.0:1769076554.864010:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076554.864013:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076554.864033:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076554.864047:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961371520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076554.864058:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076554.864095:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076554/real 1769076554] req@ffff895e44e28a80 x1855008961371264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076554.864109:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961371264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076554.864117:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076554.864120:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076554.864123:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961371264:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076554.864142:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961371392:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076554.864158:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076554.864165:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076554/real 1769076554] req@ffff895e44e29500 x1855008961371392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076554.864173:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961371392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076554.864179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076554.864181:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076554.864183:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961371392:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076554.864194:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961371520:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076554.864208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076554.864214:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076554/real 1769076554] req@ffff895e44e29180 x1855008961371520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076554.864222:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961371520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076554.864227:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076554.864229:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076554.864231:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961371520:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076555.375876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076555.567904:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076556.399903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076556.591892:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076557.423864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076557.615922:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076558.447919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076558.639988:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076559.472314:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076559.663893:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076559.983887:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076559.983896:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076559.983900:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076559.983908:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2242 00000100:00080000:1.0:1769076559.983918:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076559.983925:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076559.983968:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961371648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076559.983990:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076559.984007:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076559.984009:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076559.984011:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2242 00000100:00080000:1.0:1769076559.984015:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076559.984019:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076559.984027:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076559.984037:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961371776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076559.984046:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076559.984048:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961371648:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769076559.984050:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076559.984053:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076559.984060:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2242 00000100:00080000:1.0:1769076559.984064:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076559.984066:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076559.984069:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076559.984078:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961371904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076559.984122:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076559.984164:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076559/real 1769076559] req@ffff895e4713ea00 x1855008961371648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076559.984186:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961371648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076559.984197:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076559.984201:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076559.984206:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961371648:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076559.984231:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961371776:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076559.984255:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076559.984268:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076559/real 1769076559] req@ffff895e4713f100 x1855008961371776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076559.984289:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961371776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076559.984298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076559.984301:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076559.984305:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961371776:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076559.984311:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961371904:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076559.984332:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076559.984344:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076559/real 1769076559] req@ffff895e4713e300 x1855008961371904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076559.984357:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961371904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076559.984365:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076559.984369:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076559.984373:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961371904:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076560.495856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076560.687901:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076561.519875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076561.711980:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076562.543885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076562.735948:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076563.567822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076563.759906:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076564.591899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076564.783900:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076565.103878:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076565.103893:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076565.103899:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076565.103912:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2247 00000100:00080000:1.0:1769076565.103927:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076565.103935:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076565.104007:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961372032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076565.104042:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076565.104047:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076565.104049:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076565.104054:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2247 00000100:00080000:1.0:1769076565.104060:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076565.104066:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076565.104079:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076565.104081:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961372032:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076565.104098:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961372160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076565.104112:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076565.104117:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076565.104120:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076565.104124:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2247 00000100:00080000:1.0:1769076565.104130:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076565.104134:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076565.104140:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076565.104154:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961372288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076565.104157:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076565.104199:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076565/real 1769076565] req@ffff895e4713d880 x1855008961372032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076565.104215:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961372032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076565.104234:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076565.104238:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076565.104242:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961372032:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076565.104266:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961372160:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076565.104289:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076565.104299:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076565/real 1769076565] req@ffff895e4713ed80 x1855008961372160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076565.104307:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961372160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076565.104313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076565.104315:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076565.104317:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961372160:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076565.104321:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961372288:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076565.104332:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076565.104338:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076565/real 1769076565] req@ffff895e4713c000 x1855008961372288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076565.104345:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961372288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076565.104350:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076565.104352:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076565.104353:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961372288:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076565.615870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076565.807960:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076566.639854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076566.831908:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076567.663810:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076567.855903:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076568.687823:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076568.879862:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076569.711994:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076569.904055:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076570.223972:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076570.223991:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076570.223999:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076570.224041:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2252 00000100:00080000:3.0:1769076570.224059:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076570.224071:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076570.224132:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961372416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076570.224170:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076570.224174:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076570.224177:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076570.224183:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2252 00000100:00080000:3.0:1769076570.224189:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076570.224196:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076570.224210:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076570.224224:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961372416:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076570.224227:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961372544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076570.224250:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076570.224254:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076570.224256:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076570.224273:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2252 00000100:00080000:3.0:1769076570.224279:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076570.224283:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076570.224314:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076570.224354:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961372672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076570.224362:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076570.224488:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076570/real 1769076570] req@ffff895e453d5500 x1855008961372416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076570.224530:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961372416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076570.224552:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076570.224560:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076570.224566:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961372416:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076570.224616:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961372544:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076570.224654:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076570.224671:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076570/real 1769076570] req@ffff895e453d7b80 x1855008961372544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076570.224785:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961372544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076570.224799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076570.224804:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076570.224821:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961372544:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076570.224844:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961372672:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076570.224883:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076570.224898:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076570/real 1769076570] req@ffff895e453d7100 x1855008961372672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076570.224921:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961372672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076570.224932:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076570.224938:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076570.224944:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961372672:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076570.735859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076570.927971:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076571.760036:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076571.951956:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076572.783908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076572.976029:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076573.807954:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076573.999924:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076574.831822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076575.024035:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076575.343898:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076575.343911:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076575.343918:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076575.343931:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2257 00000100:00080000:0.0:1769076575.343945:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076575.343954:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076575.344007:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961372800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076575.344044:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076575.344048:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076575.344050:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076575.344074:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2257 00000100:00080000:0.0:1769076575.344079:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076575.344084:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076575.344097:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076575.344112:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961372928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076575.344120:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961372800:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076575.344133:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076575.344138:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076575.344139:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076575.344144:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2257 00000100:00080000:0.0:1769076575.344147:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076575.344151:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076575.344156:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076575.344166:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961373056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076575.344247:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076575.344302:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076575/real 1769076575] req@ffff895e4383e300 x1855008961372800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076575.344324:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961372800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076575.344336:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076575.344341:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076575.344345:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961372800:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076575.344387:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961372928:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076575.344413:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076575.344426:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076575/real 1769076575] req@ffff895e4383f800 x1855008961372928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076575.344438:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961372928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076575.344447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076575.344450:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076575.344454:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961372928:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076575.344460:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961373056:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076575.344502:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076575.344520:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076575/real 1769076575] req@ffff895e4383d500 x1855008961373056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076575.344539:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961373056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076575.344552:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076575.344557:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076575.344562:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961373056:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076575.855889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076576.048058:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076576.879841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076577.071912:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076577.903833:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076578.095941:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076578.927913:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076579.119890:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076579.951903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076580.143939:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076580.463937:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076580.463949:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076580.463954:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076580.463965:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2263 00000100:00080000:2.0:1769076580.463977:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076580.463985:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076580.464024:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961373184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076580.464061:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076580.464082:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076580.464085:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076580.464091:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2263 00000100:00080000:2.0:1769076580.464097:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076580.464103:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076580.464110:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961373184:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769076580.464115:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076580.464137:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961373312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076580.464150:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076580.464154:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076580.464156:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076580.464161:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2263 00000100:00080000:2.0:1769076580.464167:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076580.464173:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076580.464178:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076580.464180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076580.464197:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961373440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076580.464229:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076580/real 1769076580] req@ffff895e44e28e00 x1855008961373184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076580.464249:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961373184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076580.464259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076580.464264:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076580.464268:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961373184:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076580.464294:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961373312:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076580.464312:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076580.464322:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076580/real 1769076580] req@ffff895e44e2b480 x1855008961373312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076580.464341:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961373312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076580.464347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076580.464349:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076580.464351:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961373312:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076580.464355:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961373440:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076580.464368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076580.464375:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076580/real 1769076580] req@ffff895e44e29f80 x1855008961373440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076580.464382:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961373440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076580.464387:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076580.464389:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076580.464391:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961373440:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076580.975888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076581.168009:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076581.999846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076582.191905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076583.023877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076583.215933:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076584.047920:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076584.239996:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076585.071897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076585.263914:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076585.583926:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076585.583950:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076585.583959:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076585.583979:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2268 00000100:00080000:1.0:1769076585.583998:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076585.584024:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076585.584087:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961373568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076585.584131:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076585.584137:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076585.584140:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076585.584145:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2268 00000100:00080000:1.0:1769076585.584150:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076585.584155:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076585.584169:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076585.584222:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961373696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076585.584237:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076585.584242:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076585.584245:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076585.584249:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2268 00000100:00080000:1.0:1769076585.584255:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076585.584260:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076585.584268:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076585.584277:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961373568:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076585.584290:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961373824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076585.584440:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076585.584505:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076585/real 1769076585] req@ffff895e4713fb80 x1855008961373568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076585.584532:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961373568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076585.584568:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076585.584574:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076585.584580:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961373568:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076585.584592:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961373696:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076585.584626:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076585.584643:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076585/real 1769076585] req@ffff895e4713c700 x1855008961373696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076585.584658:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961373696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076585.584669:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076585.584674:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076585.584678:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961373696:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076585.584713:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961373824:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076585.584733:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076585.584744:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076585/real 1769076585] req@ffff895e4713c000 x1855008961373824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076585.584803:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961373824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076585.584814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076585.584818:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076585.584823:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961373824:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076586.095962:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076586.288122:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076587.119912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076587.311945:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076588.143969:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076588.335889:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076589.167871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076589.360032:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076590.191967:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076590.383977:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076590.703883:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076590.703913:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076590.703918:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076590.703930:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2273 00000100:00080000:2.0:1769076590.703942:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076590.703950:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076590.703990:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961373952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076590.704024:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076590.704029:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076590.704031:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076590.704036:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2273 00000100:00080000:2.0:1769076590.704041:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076590.704046:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076590.704057:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076590.704072:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961373952:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076590.704075:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961374080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076590.704091:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076590.704095:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076590.704098:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076590.704102:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2273 00000100:00080000:2.0:1769076590.704107:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076590.704112:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076590.704118:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076590.704131:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961374208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076590.704150:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076590.704193:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076590/real 1769076590] req@ffff895e44e2b100 x1855008961373952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076590.704210:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961373952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076590.704219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076590.704222:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076590.704226:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961373952:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076590.704251:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961374080:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076590.704279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076590.704294:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076590/real 1769076590] req@ffff895e44e2a300 x1855008961374080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076590.704309:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961374080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076590.704320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076590.704324:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076590.704329:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961374080:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076590.704352:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961374208:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076590.704374:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076590.704389:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076590/real 1769076590] req@ffff895e44e2aa00 x1855008961374208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076590.704407:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961374208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076590.704414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076590.704417:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076590.704419:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961374208:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076591.215830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076591.407879:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076592.239816:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076592.431864:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076593.263863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076593.455913:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076594.287837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076594.479866:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076595.311882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076595.503956:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076595.823964:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076595.823977:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076595.823981:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076595.823992:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2278 00000100:00080000:3.0:1769076595.824004:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076595.824012:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076595.824053:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961374336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076595.824093:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076595.824097:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076595.824099:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076595.824104:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2278 00000100:00080000:3.0:1769076595.824107:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076595.824111:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076595.824120:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076595.824132:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961374464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076595.824178:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961374336:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076595.824185:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076595.824191:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076595.824193:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076595.824199:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2278 00000100:00080000:3.0:1769076595.824205:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076595.824209:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076595.824214:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076595.824280:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961374592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076595.824283:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076595.824368:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076595/real 1769076595] req@ffff895e453d4700 x1855008961374336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076595.824391:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961374336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076595.824402:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076595.824408:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076595.824412:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961374336:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076595.824439:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961374464:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076595.824467:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076595.824480:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076595/real 1769076595] req@ffff895e453d5880 x1855008961374464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076595.824499:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961374464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076595.824512:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076595.824517:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076595.824522:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961374464:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076595.824532:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961374592:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076595.824565:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076595.824584:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076595/real 1769076595] req@ffff895e453d5180 x1855008961374592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076595.824603:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961374592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076595.824637:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076595.824643:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076595.824649:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961374592:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076596.335882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076596.527961:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076597.359820:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076597.551998:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076598.383936:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076598.575873:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076599.407929:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076599.599931:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076600.431839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076600.623967:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076600.943969:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076600.943995:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076600.944002:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076600.944017:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2283 00000100:00080000:2.0:1769076600.944031:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076600.944041:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076600.944098:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961374720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076600.944155:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076600.944182:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076600.944186:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076600.944217:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2283 00000100:00080000:2.0:1769076600.944222:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076600.944226:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076600.944236:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076600.944252:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961374848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076600.944265:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076600.944277:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961374720:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076600.944291:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076600.944294:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076600.944303:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2283 00000100:00080000:2.0:1769076600.944309:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076600.944312:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076600.944316:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076600.944333:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961374976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076600.944484:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961374848:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076600.944531:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961374976:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076601.456019:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076601.648063:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076602.479889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076602.672035:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076603.503890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076603.695969:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769076604.016246:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769076604.016317:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076604.016345:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769076604.016417:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769076604.016431:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769076604.016488:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076600/real 1769076604] req@ffff895e44e28000 x1855008961374848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076604.016534:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961374848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076604.016554:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076604.016564:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076604.016570:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961374848:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076604.016581:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076600/real 1769076604] req@ffff895e44e28380 x1855008961374976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076604.016595:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961374976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076604.016626:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076604.016630:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076604.016633:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961374976:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076604.016641:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076600/real 1769076604] req@ffff895e44e29880 x1855008961374720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076604.016653:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961374720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076604.016659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076604.016663:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076604.016666:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961374720:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076604.527912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076604.720487:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076605.551925:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076605.744017:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076606.063983:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076606.063998:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076606.064003:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076606.064015:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2288 00000100:00080000:2.0:1769076606.064029:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076606.064037:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076606.064112:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961375104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076606.064149:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076606.064153:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076606.064156:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076606.064160:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2288 00000100:00080000:2.0:1769076606.064165:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076606.064169:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076606.064183:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076606.064228:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961375232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076606.064230:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961375104:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076606.064257:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076606.064261:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076606.064264:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076606.064271:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2288 00000100:00080000:2.0:1769076606.064278:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076606.064282:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076606.064288:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076606.064307:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961375360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076606.064343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076606.064403:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076606/real 1769076606] req@ffff895e44e29880 x1855008961375104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076606.064438:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961375104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076606.064456:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076606.064463:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076606.064471:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961375104:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076606.064529:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961375232:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076606.064575:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076606.064599:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076606/real 1769076606] req@ffff895e44e28380 x1855008961375232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076606.064622:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961375232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076606.064637:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076606.064643:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076606.064649:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961375232:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076606.064678:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961375360:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076606.064713:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076606.064733:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076606/real 1769076606] req@ffff895e44e28000 x1855008961375360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076606.064821:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961375360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076606.064836:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076606.064842:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076606.064848:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961375360:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076606.575856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076606.768048:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076607.599895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076607.791922:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076608.623954:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076608.816049:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076609.647914:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076609.839924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076610.671957:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076610.863938:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076611.184616:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076611.184662:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076611.184670:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076611.184689:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2293 00000100:00080000:0.0:1769076611.184710:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076611.184725:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076611.184838:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961375488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076611.184896:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076611.184903:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076611.184906:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076611.184914:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2293 00000100:00080000:0.0:1769076611.184922:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076611.184930:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076611.184946:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076611.184979:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961375616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076611.185022:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076611.185029:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076611.185032:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076611.185039:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2293 00000100:00080000:0.0:1769076611.185046:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076611.185054:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076611.185079:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076611.185102:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961375744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076611.185210:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961375488:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769076611.185339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076611.185411:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076611/real 1769076611] req@ffff895e4383dc00 x1855008961375488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076611.185448:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961375488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076611.185469:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076611.185476:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076611.185484:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961375488:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076611.185497:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961375616:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076611.185532:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076611.185553:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076611/real 1769076611] req@ffff895e4383ea00 x1855008961375616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076611.185593:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961375616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076611.185630:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076611.185638:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076611.185646:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961375616:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076611.185659:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961375744:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076611.185693:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076611.185709:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076611/real 1769076611] req@ffff895e4383ca80 x1855008961375744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076611.185724:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961375744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076611.185734:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076611.185738:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076611.185742:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961375744:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076611.695924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076611.887974:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076612.719879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076612.912060:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076613.743893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076613.936037:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076614.767996:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076614.959982:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076615.791836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076615.984007:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076616.303994:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076616.304009:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076616.304015:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076616.304030:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2298 00000100:00080000:0.0:1769076616.304048:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076616.304058:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076616.304120:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961375872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076616.304164:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076616.304169:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076616.304172:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076616.304177:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2298 00000100:00080000:0.0:1769076616.304181:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076616.304186:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076616.304199:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076616.304226:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961376000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076616.304242:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076616.304289:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076616.304293:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076616.304293:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961375872:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076616.304309:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2298 00000100:00080000:0.0:1769076616.304316:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076616.304321:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076616.304330:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076616.304352:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961376128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076616.304416:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076616.304479:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076616/real 1769076616] req@ffff895e4383ce00 x1855008961375872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076616.304506:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961375872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076616.304542:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076616.304548:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076616.304554:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961375872:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076616.304565:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961376000:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076616.304599:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076616.304615:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076616/real 1769076616] req@ffff895e4383c000 x1855008961376000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076616.304631:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961376000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076616.304643:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076616.304647:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076616.304651:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961376000:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076616.304685:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961376128:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076616.304705:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076616.304717:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076616/real 1769076616] req@ffff895e4383f480 x1855008961376128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076616.304731:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961376128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076616.304800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076616.304804:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076616.304809:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961376128:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076616.815835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076617.008106:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076617.839861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076618.031963:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076618.863917:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076619.056032:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076619.887915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076620.079948:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076620.911935:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076621.104082:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076621.423898:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076621.423911:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076621.423917:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076621.423931:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2303 00000100:00080000:0.0:1769076621.423946:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076621.423956:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076621.424007:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961376256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076621.424039:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076621.424043:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076621.424045:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076621.424049:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2303 00000100:00080000:0.0:1769076621.424054:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076621.424058:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076621.424088:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076621.424104:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961376384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076621.424118:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076621.424121:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076621.424124:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076621.424129:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2303 00000100:00080000:0.0:1769076621.424138:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076621.424138:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961376256:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076621.424154:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076621.424159:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076621.424173:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961376512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076621.424237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076621.424287:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076621/real 1769076621] req@ffff895e4383fb80 x1855008961376256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076621.424303:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961376256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076621.424313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076621.424317:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076621.424320:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961376256:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076621.424346:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961376384:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076621.424362:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076621.424370:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076621/real 1769076621] req@ffff895e4383c380 x1855008961376384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076621.424378:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961376384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076621.424384:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076621.424387:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076621.424389:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961376384:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076621.424393:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961376512:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076621.424403:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076621.424409:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076621/real 1769076621] req@ffff895e4383f100 x1855008961376512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076621.424440:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961376512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076621.424446:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076621.424449:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076621.424451:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961376512:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076621.935974:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076622.127958:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076622.960014:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076623.152124:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076623.983889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076624.175998:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076625.007954:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076625.200031:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076626.031923:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076626.224057:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076626.543882:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076626.543892:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076626.543897:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076626.543908:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2309 00000100:00080000:0.0:1769076626.543920:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076626.543927:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076626.543966:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961376640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076626.543994:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076626.543997:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076626.544013:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076626.544019:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2309 00000100:00080000:0.0:1769076626.544027:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076626.544033:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076626.544044:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076626.544051:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961376640:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076626.544063:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961376768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076626.544075:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076626.544079:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076626.544081:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076626.544084:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2309 00000100:00080000:0.0:1769076626.544087:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076626.544090:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076626.544094:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076626.544106:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961376896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076626.544139:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076626.544179:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076626/real 1769076626] req@ffff895e4383e680 x1855008961376640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076626.544196:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961376640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076626.544205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076626.544209:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076626.544213:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961376640:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076626.544252:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961376768:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076626.544271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076626.544280:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076626/real 1769076626] req@ffff895e4383d880 x1855008961376768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076626.544299:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961376768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076626.544306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076626.544309:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076626.544311:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961376768:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076626.544316:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961376896:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076626.544331:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076626.544339:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076626/real 1769076626] req@ffff895e4383c700 x1855008961376896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076626.544347:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961376896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076626.544353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076626.544355:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076626.544358:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961376896:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076627.055828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076627.247923:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076628.079856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076628.271978:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076629.103900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076629.295974:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076630.127895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076630.319980:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076631.151859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076631.343915:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076631.663866:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076631.663877:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076631.663882:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076631.663892:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2314 00000100:00080000:1.0:1769076631.663904:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076631.663933:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076631.663978:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961377024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076631.664014:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076631.664017:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076631.664018:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076631.664022:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2314 00000100:00080000:1.0:1769076631.664026:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076631.664029:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076631.664038:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076631.664050:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961377152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076631.664058:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076631.664061:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076631.664062:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076631.664065:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2314 00000100:00080000:1.0:1769076631.664068:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076631.664071:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076631.664074:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076631.664083:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961377280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076631.664091:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961377024:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769076631.664202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076631.664266:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076631/real 1769076631] req@ffff895e4713ed80 x1855008961377024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076631.664304:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961377024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076631.664333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076631.664339:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076631.664344:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961377024:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076631.664353:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961377152:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076631.664378:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076631.664392:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076631/real 1769076631] req@ffff895e4713d880 x1855008961377152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076631.664406:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961377152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076631.664416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076631.664419:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076631.664423:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961377152:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076631.664452:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961377280:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076631.664472:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076631.664485:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076631/real 1769076631] req@ffff895e4713e300 x1855008961377280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076631.664498:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961377280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076631.664519:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076631.664524:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076631.664528:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961377280:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076632.175823:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076632.367879:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076633.199874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076633.392038:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076634.223910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076634.416078:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076635.247876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076635.440029:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076636.271906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076636.463979:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076636.783901:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076636.783910:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076636.783914:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076636.783922:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2319 00000100:00080000:3.0:1769076636.783932:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076636.783938:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076636.783982:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961377408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076636.784007:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076636.784010:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076636.784011:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076636.784014:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2319 00000100:00080000:3.0:1769076636.784016:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076636.784018:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076636.784025:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076636.784034:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961377536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076636.784042:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076636.784053:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961377408:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076636.784061:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076636.784062:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076636.784066:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2319 00000100:00080000:3.0:1769076636.784070:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076636.784073:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076636.784087:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076636.784097:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961377664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076636.784126:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076636.784161:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076636/real 1769076636] req@ffff895e453d6680 x1855008961377408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076636.784177:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961377408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076636.784196:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076636.784201:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076636.784205:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961377408:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076636.784213:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961377536:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076636.784239:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076636.784249:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076636/real 1769076636] req@ffff895e453d4380 x1855008961377536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076636.784258:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961377536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076636.784263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076636.784266:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076636.784268:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961377536:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076636.784287:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961377664:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076636.784299:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076636.784305:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076636/real 1769076636] req@ffff895e453d6d80 x1855008961377664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076636.784312:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961377664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076636.784318:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076636.784319:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076636.784321:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961377664:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076637.295844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076637.487997:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076638.319882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076638.511994:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076639.343812:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076639.535954:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076640.367897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076640.559887:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076641.391867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076641.583950:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076641.903893:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076641.903912:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076641.903918:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076641.903933:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2324 00000100:00080000:3.0:1769076641.903951:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076641.903962:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076641.904026:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961377792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076641.904076:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076641.904083:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076641.904087:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076641.904096:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2324 00000100:00080000:3.0:1769076641.904103:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076641.904111:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076641.904192:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076641.904208:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961377792:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076641.904222:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961377920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076641.904242:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076641.904247:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076641.904250:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076641.904255:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2324 00000100:00080000:3.0:1769076641.904262:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076641.904268:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076641.904285:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076641.904301:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961378048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076641.904341:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076641.904405:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076641/real 1769076641] req@ffff895e453d6300 x1855008961377792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076641.904431:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961377792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076641.904447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076641.904453:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076641.904458:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961377792:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076641.904526:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961377920:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076641.904561:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076641.904583:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076641/real 1769076641] req@ffff895e453d6d80 x1855008961377920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076641.904608:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961377920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076641.904625:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076641.904632:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076641.904639:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961377920:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076641.904652:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961378048:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076641.904717:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076641.904735:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076641/real 1769076641] req@ffff895e453d4380 x1855008961378048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076641.904827:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961378048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076641.904843:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076641.904850:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076641.904857:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961378048:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076642.415843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076642.607981:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076643.439919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076643.631906:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076644.463958:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076644.655993:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076645.487882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076645.679947:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076646.511902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076646.703879:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076647.023855:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076647.023879:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076647.023883:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076647.023892:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2329 00000100:00080000:1.0:1769076647.023901:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076647.023908:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076647.023959:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961378176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076647.024006:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076647.024009:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076647.024010:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076647.024013:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2329 00000100:00080000:1.0:1769076647.024015:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076647.024018:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076647.024025:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076647.024034:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961378304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076647.024041:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076647.024058:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076647.024060:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076647.024063:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2329 00000100:00100000:0.0:1769076647.024069:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961378176:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769076647.024069:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076647.024080:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076647.024086:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076647.024104:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961378432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076647.024131:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076647.024165:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076647/real 1769076647] req@ffff895e4713f100 x1855008961378176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076647.024179:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961378176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076647.024186:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076647.024189:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076647.024193:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961378176:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076647.024198:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961378304:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076647.024233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076647.024245:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076647/real 1769076647] req@ffff895e4713ea00 x1855008961378304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076647.024289:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961378304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076647.024296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076647.024298:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076647.024300:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961378304:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076647.024321:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961378432:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076647.024335:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076647.024342:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076647/real 1769076647] req@ffff895e4713d500 x1855008961378432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076647.024350:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961378432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076647.024356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076647.024357:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076647.024360:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961378432:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076647.535834:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076647.727883:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076648.559831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076648.751880:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076649.583884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076649.775888:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076650.607846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076650.799892:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076651.631891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076651.823943:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076652.143938:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076652.143954:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076652.143959:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076652.143972:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2334 00000100:00080000:0.0:1769076652.143988:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076652.143997:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076652.144061:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961378560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076652.144095:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076652.144098:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076652.144099:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076652.144103:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2334 00000100:00080000:0.0:1769076652.144106:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076652.144110:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076652.144120:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076652.144128:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961378688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076652.144135:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076652.144137:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076652.144138:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076652.144141:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2334 00000100:00080000:0.0:1769076652.144142:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076652.144145:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076652.144148:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076652.144154:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961378816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076652.144186:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961378560:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769076652.144270:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076652.144343:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076652/real 1769076652] req@ffff895e4383df80 x1855008961378560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076652.144361:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961378560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076652.144370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076652.144373:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076652.144377:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961378560:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076652.144383:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961378688:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076652.144411:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076652.144421:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076652/real 1769076652] req@ffff895e4383ed80 x1855008961378688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076652.144430:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961378688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076652.144436:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076652.144439:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076652.144441:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961378688:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076652.144445:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961378816:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076652.144456:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076652.144462:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076652/real 1769076652] req@ffff895e4383d180 x1855008961378816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076652.144469:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961378816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076652.144474:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076652.144476:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076652.144478:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961378816:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076652.655829:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076652.847929:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076653.679854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076653.871950:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076654.703874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076654.895950:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076655.727901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076655.919926:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076656.751844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076656.943992:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076657.263880:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076657.263895:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076657.263901:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076657.263913:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2339 00000100:00080000:3.0:1769076657.263925:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076657.263956:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076657.264001:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961378944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076657.264038:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076657.264042:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076657.264043:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076657.264048:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2339 00000100:00080000:3.0:1769076657.264054:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076657.264059:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076657.264071:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076657.264076:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961378944:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076657.264090:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961379072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076657.264105:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076657.264108:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076657.264110:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076657.264114:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2339 00000100:00080000:3.0:1769076657.264117:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076657.264121:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076657.264126:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076657.264137:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961379200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076657.264165:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076657.264206:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076657/real 1769076657] req@ffff895e453d6680 x1855008961378944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076657.264241:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961378944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076657.264254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076657.264258:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076657.264262:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961378944:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076657.264288:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961379072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076657.264306:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076657.264316:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076657/real 1769076657] req@ffff895e453d5180 x1855008961379072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076657.264325:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961379072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076657.264331:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076657.264333:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076657.264335:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961379072:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076657.264339:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961379200:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076657.264349:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076657.264355:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076657/real 1769076657] req@ffff895e453d5880 x1855008961379200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076657.264373:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961379200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076657.264379:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076657.264381:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076657.264383:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961379200:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076657.775850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076657.967979:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076658.799821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076658.991966:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076659.823873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076660.015905:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076660.847889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076661.039925:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076661.871859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076662.063914:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076662.383849:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076662.383859:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076662.383863:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076662.383871:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2344 00000100:00080000:0.0:1769076662.383882:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076662.383889:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076662.383938:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961379328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076662.383971:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076662.383974:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076662.383975:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076662.383978:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2344 00000100:00080000:0.0:1769076662.383982:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076662.383985:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076662.383992:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076662.384003:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961379456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076662.384020:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961379328:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076662.384037:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076662.384040:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076662.384043:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076662.384048:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2344 00000100:00080000:0.0:1769076662.384053:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076662.384058:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076662.384064:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076662.384083:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076662.384088:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961379584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076662.384154:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076662/real 1769076662] req@ffff895e4383d180 x1855008961379328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076662.384171:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961379328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076662.384179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076662.384182:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076662.384185:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961379328:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076662.384205:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961379456:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076662.384222:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076662.384230:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076662/real 1769076662] req@ffff895e4383ed80 x1855008961379456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076662.384238:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961379456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076662.384244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076662.384246:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076662.384249:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961379456:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076662.384252:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961379584:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076662.384263:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076662.384269:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076662/real 1769076662] req@ffff895e4383df80 x1855008961379584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076662.384276:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961379584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076662.384291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076662.384293:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076662.384295:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961379584:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076662.895849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076663.087950:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076663.919835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076664.111891:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076664.943840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076665.135891:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076665.967835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076666.159881:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076666.991872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076667.183953:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076667.503872:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076667.503884:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076667.503889:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076667.503900:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2350 00000100:00080000:0.0:1769076667.503912:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076667.503922:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076667.503959:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961379712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076667.503989:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076667.503993:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076667.503995:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076667.503999:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2350 00000100:00080000:0.0:1769076667.504019:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076667.504025:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076667.504035:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076667.504074:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961379712:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076667.504080:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961379840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076667.504099:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076667.504103:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076667.504106:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076667.504111:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2350 00000100:00080000:0.0:1769076667.504116:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076667.504120:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076667.504126:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076667.504153:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961379968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076667.504232:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961379840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076667.504265:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961379968:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076668.015845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076668.207957:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076669.039886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076669.231951:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076670.063836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076670.255958:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769076670.576081:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:0.0:1769076670.576132:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769076670.576150:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:0.0:1769076670.576190:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:0.0:1769076670.576197:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769076670.576252:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076667/real 1769076670] req@ffff895e4383d880 x1855008961379840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076670.576277:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961379840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076670.576288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076670.576292:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076670.576297:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961379840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076670.576303:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076667/real 1769076670] req@ffff895e4383e680 x1855008961379968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076670.576311:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961379968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076670.576316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076670.576319:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076670.576340:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961379968:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076670.576346:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076667/real 1769076670] req@ffff895e4383c700 x1855008961379712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076670.576355:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961379712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076670.576360:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076670.576373:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076670.576376:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961379712:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076671.087884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076671.279916:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076672.111923:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076672.304001:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076672.623923:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076672.623937:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076672.623944:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076672.623975:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2355 00000100:00080000:1.0:1769076672.624024:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076672.624031:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076672.624081:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961380096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076672.624111:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076672.624114:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076672.624116:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076672.624120:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2355 00000100:00080000:1.0:1769076672.624123:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076672.624127:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076672.624137:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076672.624150:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961380224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076672.624163:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076672.624166:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076672.624168:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076672.624175:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961380096:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769076672.624176:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2355 00000100:00080000:1.0:1769076672.624183:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076672.624190:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076672.624194:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076672.624207:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961380352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076672.624264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076672.624324:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076672/real 1769076672] req@ffff895e4713ce00 x1855008961380096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076672.624346:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961380096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076672.624358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076672.624363:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076672.624368:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961380096:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076672.624396:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961380224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076672.624431:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076672.624447:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076672/real 1769076672] req@ffff895e4713f480 x1855008961380224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076672.624483:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961380224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076672.624493:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076672.624497:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076672.624500:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961380224:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076672.624507:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961380352:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076672.624534:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076672.624559:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076672/real 1769076672] req@ffff895e4713ca80 x1855008961380352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076672.624574:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961380352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076672.624583:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076672.624586:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076672.624590:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961380352:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076673.135857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076673.327912:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076674.159890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076674.351903:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076675.183872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076675.375944:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076676.207849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076676.399961:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076677.232010:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076677.424069:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076677.744017:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076677.744069:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076677.744075:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076677.744089:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2360 00000100:00080000:0.0:1769076677.744104:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076677.744113:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076677.744173:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961380480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076677.744208:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076677.744211:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076677.744213:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076677.744217:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2360 00000100:00080000:0.0:1769076677.744220:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076677.744223:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076677.744239:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076677.744254:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961380608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076677.744265:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076677.744268:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076677.744270:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076677.744273:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2360 00000100:00080000:0.0:1769076677.744278:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076677.744282:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076677.744286:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076677.744294:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961380736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076677.744343:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961380480:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769076677.744475:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076677.744541:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076677/real 1769076677] req@ffff895e4383f100 x1855008961380480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076677.744580:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961380480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076677.744619:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076677.744637:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076677.744655:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961380480:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076677.744666:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961380608:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076677.744703:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076677.744726:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076677/real 1769076677] req@ffff895e4383c380 x1855008961380608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076677.744813:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961380608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076677.744831:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076677.744836:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076677.744840:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961380608:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076677.744912:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961380736:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076677.744956:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076677.744981:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076677/real 1769076677] req@ffff895e4383fb80 x1855008961380736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076677.745004:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961380736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076677.745014:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076677.745018:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076677.745022:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961380736:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076678.255930:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076678.448016:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076679.279920:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076679.471951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076680.303877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076680.495945:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076681.327912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076681.519948:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076682.351926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076682.544048:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076682.863947:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076682.863961:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076682.863966:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076682.863978:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2365 00000100:00080000:1.0:1769076682.863993:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076682.864002:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076682.864050:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961380864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076682.864092:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076682.864096:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076682.864098:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076682.864102:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2365 00000100:00080000:1.0:1769076682.864105:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076682.864108:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076682.864118:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076682.864131:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961380992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076682.864142:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076682.864146:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076682.864148:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076682.864156:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961380864:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769076682.864176:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2365 00000100:00080000:1.0:1769076682.864181:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076682.864186:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076682.864192:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076682.864205:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961381120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076682.864249:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076682.864295:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076682/real 1769076682] req@ffff895e4713f800 x1855008961380864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076682.864315:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961380864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076682.864325:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076682.864330:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076682.864334:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961380864:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076682.864363:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961380992:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076682.864382:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076682.864393:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076682/real 1769076682] req@ffff895e4713dc00 x1855008961380992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076682.864403:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961380992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076682.864411:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076682.864415:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076682.864418:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961380992:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076682.864423:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961381120:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076682.864436:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076682.864443:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076682/real 1769076682] req@ffff895e4713d180 x1855008961381120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076682.864463:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961381120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076682.864470:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076682.864473:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076682.864475:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961381120:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076683.375926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076683.567975:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076684.400002:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076684.592035:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076685.423940:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076685.616022:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076686.447953:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076686.639971:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076687.471873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076687.663969:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076687.983867:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076687.983877:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076687.983881:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076687.983892:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2370 00000100:00080000:1.0:1769076687.983903:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076687.983911:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076687.983947:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961381248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076687.983977:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076687.983980:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076687.983982:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076687.983986:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2370 00000100:00080000:1.0:1769076687.983990:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076687.983994:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076687.984003:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076687.984014:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961381376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076687.984040:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076687.984044:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961381248:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769076687.984045:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076687.984050:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076687.984057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2370 00000100:00080000:1.0:1769076687.984062:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076687.984067:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076687.984072:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076687.984088:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961381504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076687.984116:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076687.984165:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076687/real 1769076687] req@ffff895e4713df80 x1855008961381248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076687.984186:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961381248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076687.984198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076687.984202:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076687.984206:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961381248:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076687.984230:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961381376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076687.984253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076687.984266:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076687/real 1769076687] req@ffff895e4713c380 x1855008961381376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076687.984289:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961381376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076687.984295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076687.984297:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076687.984300:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961381376:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076687.984304:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961381504:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076687.984319:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076687.984326:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076687/real 1769076687] req@ffff895e4713d180 x1855008961381504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076687.984334:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961381504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076687.984338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076687.984340:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076687.984342:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961381504:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076688.495894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076688.687950:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076689.519832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076689.711888:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076690.543904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076690.735881:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076691.567828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076691.759857:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076692.591883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076692.783950:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076693.103862:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076693.103875:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076693.103882:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076693.103894:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2375 00000100:00080000:1.0:1769076693.103908:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076693.103917:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076693.103969:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961381632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076693.104019:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076693.104026:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076693.104030:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076693.104039:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2375 00000100:00080000:1.0:1769076693.104071:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076693.104082:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076693.104099:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076693.104103:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961381632:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076693.104127:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961381760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076693.104147:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076693.104152:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076693.104155:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076693.104160:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2375 00000100:00080000:1.0:1769076693.104167:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076693.104171:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076693.104177:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076693.104197:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769076693.104198:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961381888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076693.104260:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076693/real 1769076693] req@ffff895e4713dc00 x1855008961381632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076693.104291:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961381632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076693.104340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076693.104349:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076693.104358:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961381632:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076693.104424:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961381760:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076693.104461:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076693.104480:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076693/real 1769076693] req@ffff895e4713f800 x1855008961381760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076693.104496:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961381760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076693.104507:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076693.104512:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076693.104516:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961381760:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076693.104524:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961381888:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076693.104543:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076693.104554:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076693/real 1769076693] req@ffff895e4713ca80 x1855008961381888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076693.104568:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961381888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076693.104578:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076693.104581:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076693.104585:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961381888:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076693.615890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076693.807986:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076694.639829:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076694.831917:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076695.663818:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076695.855873:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076696.687870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076696.879920:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076697.711860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076697.903931:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076698.223876:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076698.223884:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076698.223888:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076698.223896:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2380 00000100:00080000:3.0:1769076698.223906:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076698.223911:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076698.223943:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961382016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076698.223971:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076698.223973:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076698.223975:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076698.223978:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2380 00000100:00080000:3.0:1769076698.223980:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076698.223982:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076698.223989:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076698.223998:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961382144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076698.224005:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076698.224008:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076698.224009:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076698.224012:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2380 00000100:00080000:3.0:1769076698.224014:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076698.224016:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076698.224024:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961382016:192.168.204.117@tcp:250:kworker.0 00000008:00100000:3.0:1769076698.224036:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076698.224048:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961382272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076698.224082:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076698.224112:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076698/real 1769076698] req@ffff895e453d4700 x1855008961382016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076698.224126:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961382016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076698.224133:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076698.224136:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076698.224139:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961382016:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076698.224145:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961382144:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076698.224158:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076698.224166:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076698/real 1769076698] req@ffff895e453d7100 x1855008961382144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076698.224173:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961382144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076698.224179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076698.224181:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076698.224183:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961382144:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076698.224201:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961382272:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076698.224230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076698.224238:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076698/real 1769076698] req@ffff895e453d7b80 x1855008961382272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076698.224246:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961382272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076698.224251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076698.224253:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076698.224255:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961382272:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076698.735832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076698.927874:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076699.759867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076699.952033:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076700.783876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076700.975994:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076701.807857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076701.999923:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076702.831843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076703.023905:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076703.343874:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076703.343883:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076703.343886:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076703.343895:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2385 00000100:00080000:2.0:1769076703.343905:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076703.343910:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076703.343947:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961382400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076703.343978:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076703.343982:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076703.343983:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076703.343986:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2385 00000100:00080000:2.0:1769076703.343990:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076703.343994:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076703.344002:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076703.344014:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961382528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076703.344020:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961382400:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076703.344027:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076703.344030:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076703.344031:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076703.344034:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2385 00000100:00080000:2.0:1769076703.344036:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076703.344039:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076703.344056:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076703.344065:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961382656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076703.344080:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076703.344116:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076703/real 1769076703] req@ffff895e44e2aa00 x1855008961382400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076703.344140:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961382400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076703.344148:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076703.344151:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076703.344154:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961382400:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076703.344174:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961382528:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076703.344188:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076703.344196:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076703/real 1769076703] req@ffff895e44e2a300 x1855008961382528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076703.344229:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961382528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076703.344235:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076703.344237:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076703.344240:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961382528:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076703.344244:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961382656:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076703.344259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076703.344266:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076703/real 1769076703] req@ffff895e44e2b100 x1855008961382656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076703.344273:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961382656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076703.344278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076703.344280:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076703.344282:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961382656:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076703.855838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076704.047933:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076704.879901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076705.072068:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076705.903861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076706.096049:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076706.927972:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076707.119953:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076707.951870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076708.143889:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076708.463833:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076708.463844:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076708.463848:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076708.463859:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2391 00000100:00080000:3.0:1769076708.463872:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076708.463879:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076708.463917:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961382784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076708.463945:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076708.463948:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076708.463949:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076708.463952:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2391 00000100:00080000:3.0:1769076708.463955:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076708.463958:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076708.463967:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076708.463977:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961382912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076708.463985:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769076708.463988:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961382784:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076708.463989:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076708.463992:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076708.464000:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2391 00000100:00080000:3.0:1769076708.464020:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076708.464025:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076708.464031:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076708.464044:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961383040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076708.464054:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076708.464088:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076708/real 1769076708] req@ffff895e453d5500 x1855008961382784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076708.464101:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961382784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076708.464121:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076708.464124:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076708.464127:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961382784:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076708.464146:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961382912:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076708.464163:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076708.464171:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076708/real 1769076708] req@ffff895e453d7480 x1855008961382912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076708.464179:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961382912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076708.464194:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076708.464197:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076708.464199:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961382912:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076708.464203:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961383040:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076708.464215:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076708.464222:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076708/real 1769076708] req@ffff895e453d4e00 x1855008961383040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076708.464229:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961383040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076708.464240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076708.464242:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076708.464245:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961383040:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076708.975872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076709.167982:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076709.999934:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076710.191929:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076711.023830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076711.215900:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076712.047878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076712.239954:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076713.071836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076713.263869:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076713.583914:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076713.583924:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076713.583927:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076713.583935:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2396 00000100:00080000:1.0:1769076713.583945:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076713.583968:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076713.584008:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961383168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076713.584031:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076713.584034:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076713.584035:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076713.584038:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2396 00000100:00080000:1.0:1769076713.584041:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076713.584043:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076713.584051:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076713.584058:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961383296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076713.584065:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076713.584068:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076713.584069:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076713.584072:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2396 00000100:00080000:1.0:1769076713.584073:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076713.584076:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076713.584079:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076713.584084:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961383424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076713.584092:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961383168:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769076713.584176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076713.584217:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076713/real 1769076713] req@ffff895e4713f480 x1855008961383168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076713.584232:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961383168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076713.584240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076713.584243:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076713.584246:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961383168:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076713.584252:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961383296:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076713.584269:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076713.584294:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076713/real 1769076713] req@ffff895e4713ce00 x1855008961383296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076713.584303:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961383296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076713.584309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076713.584311:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076713.584314:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961383296:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076713.584334:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961383424:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076713.584348:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076713.584354:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076713/real 1769076713] req@ffff895e4713d500 x1855008961383424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076713.584370:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961383424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076713.584376:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076713.584378:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076713.584380:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961383424:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076714.095893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076714.287992:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076715.119831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076715.311958:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076716.143843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076716.335881:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076717.167903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076717.359951:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076718.191954:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076718.383980:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076718.703950:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076718.703963:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076718.703969:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076718.703982:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2401 00000100:00080000:0.0:1769076718.703997:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076718.704006:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076718.704090:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961383552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076718.704136:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076718.704143:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076718.704147:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076718.704155:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2401 00000100:00080000:0.0:1769076718.704162:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076718.704168:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076718.704184:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076718.704253:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961383680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076718.704273:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961383552:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076718.704305:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076718.704310:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076718.704314:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076718.704320:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2401 00000100:00080000:0.0:1769076718.704327:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076718.704334:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076718.704343:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076718.704363:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961383808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076718.704363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076718.704420:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076718/real 1769076718] req@ffff895e4383f480 x1855008961383552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076718.704452:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961383552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076718.704471:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076718.704478:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076718.704483:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961383552:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076718.704514:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961383680:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076718.704549:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076718.704566:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076718/real 1769076718] req@ffff895e4383c000 x1855008961383680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076718.704581:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961383680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076718.704603:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076718.704607:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076718.704612:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961383680:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076718.704620:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961383808:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076718.704647:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076718.704660:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076718/real 1769076718] req@ffff895e4383ce00 x1855008961383808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076718.704674:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961383808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076718.704683:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076718.704686:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076718.704690:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961383808:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076719.215942:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076719.407918:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076720.239878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076720.431905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076721.263927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076721.456010:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076722.287870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076722.479867:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076723.311858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076723.504007:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076723.823874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076723.823897:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076723.823902:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076723.823914:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2406 00000100:00080000:1.0:1769076723.823927:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076723.823934:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076723.823973:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961383936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076723.824017:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076723.824020:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076723.824023:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076723.824027:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2406 00000100:00080000:1.0:1769076723.824030:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076723.824034:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076723.824054:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076723.824068:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961384064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076723.824083:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961383936:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769076723.824084:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076723.824092:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076723.824096:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076723.824125:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2406 00000100:00080000:1.0:1769076723.824129:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076723.824133:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076723.824170:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076723.824186:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961384192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076723.824230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076723.824273:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076723/real 1769076723] req@ffff895e4713ea00 x1855008961383936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076723.824293:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961383936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076723.824304:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076723.824308:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076723.824322:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961383936:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076723.824348:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961384064:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076723.824377:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076723.824390:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076723/real 1769076723] req@ffff895e4713f100 x1855008961384064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076723.824403:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961384064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076723.824411:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076723.824415:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076723.824419:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961384064:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076723.824424:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961384192:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076723.824439:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076723.824448:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076723/real 1769076723] req@ffff895e4713e300 x1855008961384192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076723.824459:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961384192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076723.824466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076723.824469:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076723.824472:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961384192:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076724.335897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076724.527958:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076725.359940:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076725.551955:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076726.383875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076726.575865:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076727.407906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076727.599987:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076728.431830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076728.624046:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076728.943977:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076728.943990:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076728.943996:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076728.944008:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2411 00000100:00080000:0.0:1769076728.944062:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076728.944072:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076728.944117:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961384320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076728.944156:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076728.944160:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076728.944163:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076728.944167:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2411 00000100:00080000:0.0:1769076728.944171:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076728.944175:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076728.944208:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076728.944226:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961384448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076728.944237:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961384320:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076728.944246:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076728.944253:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076728.944255:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076728.944261:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2411 00000100:00080000:0.0:1769076728.944268:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076728.944273:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076728.944280:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076728.944306:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961384576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076728.944380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076728.944461:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076728/real 1769076728] req@ffff895e4383ca80 x1855008961384320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076728.944485:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961384320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076728.944499:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076728.944504:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076728.944509:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961384320:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076728.944551:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961384448:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076728.944580:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076728.944601:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076728/real 1769076728] req@ffff895e4383ea00 x1855008961384448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076728.944617:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961384448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076728.944627:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076728.944631:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076728.944635:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961384448:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076728.944654:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961384576:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076728.944675:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076728.944688:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076728/real 1769076728] req@ffff895e4383dc00 x1855008961384576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076728.944701:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961384576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076728.944746:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076728.944751:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076728.944755:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961384576:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076729.455943:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076729.648070:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076730.479837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076730.672025:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076731.503934:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076731.696011:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076732.527905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076732.720033:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076733.551826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076733.743915:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076734.063806:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076734.063817:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076734.063822:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076734.063832:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2416 00000100:00080000:0.0:1769076734.063843:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076734.063852:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076734.063890:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961384704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076734.063921:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076734.063924:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076734.063925:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076734.063942:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2416 00000100:00080000:0.0:1769076734.063961:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076734.063965:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076734.063972:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961384704:192.168.204.117@tcp:250:kworker.0 00000008:00100000:0.0:1769076734.063974:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076734.063988:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961384832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076734.063997:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076734.063999:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076734.064000:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076734.064003:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2416 00000100:00080000:0.0:1769076734.064006:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076734.064009:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076734.064011:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076734.064019:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961384960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076734.064080:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961384832:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076734.064118:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961384960:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076734.575829:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076734.767963:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076735.599890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076735.791956:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076736.623831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076736.815996:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769076737.136017:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769076737.136054:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076737.136070:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769076737.136102:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769076737.136111:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769076737.136174:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076734/real 1769076737] req@ffff895e4383f800 x1855008961384832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076737.136215:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961384832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076737.136229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076737.136235:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076737.136241:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961384832:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076737.136252:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076734/real 1769076737] req@ffff895e4383e300 x1855008961384960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076737.136272:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961384960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076737.136280:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076737.136284:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076737.136317:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961384960:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076737.136326:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076734/real 1769076737] req@ffff895e4383d500 x1855008961384704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076737.136341:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961384704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076737.136350:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076737.136355:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076737.136359:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961384704:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076737.647879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076737.839989:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076738.671919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076738.863940:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076739.183871:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076739.183886:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076739.183893:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076739.183905:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2421 00000100:00080000:0.0:1769076739.183921:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076739.183931:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076739.183983:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961385088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076739.184023:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076739.184046:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076739.184049:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076739.184054:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2421 00000100:00080000:0.0:1769076739.184059:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076739.184065:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076739.184077:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076739.184093:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961385216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076739.184106:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076739.184110:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076739.184113:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076739.184119:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961385088:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076739.184121:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2421 00000100:00080000:0.0:1769076739.184131:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076739.184138:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076739.184145:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076739.184161:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961385344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076739.184226:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076739.184281:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076739/real 1769076739] req@ffff895e4383d500 x1855008961385088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076739.184305:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961385088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076739.184316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076739.184321:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076739.184325:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961385088:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076739.184359:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961385216:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076739.184387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076739.184402:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076739/real 1769076739] req@ffff895e4383e300 x1855008961385216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076739.184437:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961385216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076739.184447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076739.184451:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076739.184455:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961385216:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076739.184463:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961385344:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076739.184488:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076739.184500:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076739/real 1769076739] req@ffff895e4383f800 x1855008961385344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076739.184512:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961385344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076739.184521:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076739.184525:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076739.184529:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961385344:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076739.695934:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076739.888006:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076740.719968:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076740.911967:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076741.743879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076741.935921:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076742.767869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076742.959893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076743.791928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076743.983924:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076744.303951:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076744.303960:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076744.303965:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076744.303975:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2426 00000100:00080000:0.0:1769076744.303985:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076744.304024:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076744.304056:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961385472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076744.304099:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076744.304103:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076744.304105:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076744.304110:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2426 00000100:00080000:0.0:1769076744.304114:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076744.304119:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076744.304128:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076744.304139:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961385600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076744.304146:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076744.304149:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076744.304152:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076744.304157:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2426 00000100:00080000:0.0:1769076744.304160:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076744.304163:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076744.304168:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076744.304175:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961385728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076744.304191:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961385472:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769076744.304275:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076744.304319:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076744/real 1769076744] req@ffff895e4383dc00 x1855008961385472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076744.304335:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961385472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076744.304361:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076744.304365:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076744.304368:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961385472:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076744.304374:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961385600:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076744.304392:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076744.304400:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076744/real 1769076744] req@ffff895e4383ea00 x1855008961385600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076744.304408:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961385600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076744.304415:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076744.304418:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076744.304420:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961385600:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076744.304443:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961385728:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076744.304454:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076744.304460:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076744/real 1769076744] req@ffff895e4383ca80 x1855008961385728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076744.304467:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961385728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076744.304472:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076744.304474:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076744.304476:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961385728:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076744.815892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076745.007972:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076745.839904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076746.031876:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076746.863835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076747.055903:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076747.887843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076748.079934:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076748.911872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076749.103930:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076749.423910:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076749.423922:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076749.423927:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076749.423937:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2431 00000100:00080000:2.0:1769076749.423947:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076749.423955:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076749.423992:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961385856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076749.424025:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076749.424030:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076749.424032:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076749.424036:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2431 00000100:00080000:2.0:1769076749.424040:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076749.424045:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076749.424055:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076749.424069:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961385984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076749.424078:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076749.424081:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076749.424096:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076749.424099:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961385856:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076749.424103:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2431 00000100:00080000:2.0:1769076749.424113:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076749.424119:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076749.424124:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076749.424146:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961386112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076749.424195:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076749.424242:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076749/real 1769076749] req@ffff895e44e29f80 x1855008961385856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076749.424259:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961385856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076749.424269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076749.424272:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076749.424276:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961385856:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076749.424283:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961385984:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076749.424298:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076749.424306:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076749/real 1769076749] req@ffff895e44e2b480 x1855008961385984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076749.424314:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961385984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076749.424319:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076749.424321:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076749.424323:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961385984:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076749.424366:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961386112:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076749.424381:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076749.424388:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076749/real 1769076749] req@ffff895e44e28e00 x1855008961386112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076749.424396:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961386112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076749.424401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076749.424403:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076749.424406:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961386112:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076749.935828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076750.127962:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076750.959857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076751.151897:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076751.983831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076752.175929:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076753.007841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076753.199895:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076754.031851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076754.223863:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076754.543930:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076754.543942:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076754.543946:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076754.543957:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2437 00000100:00080000:1.0:1769076754.543969:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076754.543977:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076754.544017:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961386240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076754.544052:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076754.544056:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076754.544058:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076754.544078:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2437 00000100:00080000:1.0:1769076754.544084:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076754.544090:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076754.544100:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961386240:192.168.204.117@tcp:250:kworker.0 00000008:00100000:1.0:1769076754.544101:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076754.544121:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961386368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076754.544135:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076754.544139:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076754.544141:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076754.544146:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2437 00000100:00080000:1.0:1769076754.544151:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076754.544155:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076754.544161:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076754.544172:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769076754.544176:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961386496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076754.544219:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076754/real 1769076754] req@ffff895e4713d880 x1855008961386240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076754.544249:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961386240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076754.544260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076754.544272:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076754.544277:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961386240:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076754.544300:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961386368:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076754.544326:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076754.544341:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076754/real 1769076754] req@ffff895e4713ed80 x1855008961386368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076754.544355:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961386368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076754.544363:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076754.544367:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076754.544371:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961386368:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076754.544377:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961386496:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076754.544397:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076754.544409:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076754/real 1769076754] req@ffff895e4713c000 x1855008961386496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076754.544422:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961386496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076754.544431:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076754.544434:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076754.544438:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961386496:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076755.055852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076755.247944:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076756.079811:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076756.271864:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076757.103845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076757.295879:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076758.127915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076758.319914:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076759.151837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076759.343926:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076759.663870:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076759.663881:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076759.663894:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076759.663904:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2442 00000100:00080000:0.0:1769076759.665059:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076759.665066:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076759.665094:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961386624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076759.665124:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076759.665128:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076759.665130:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076759.665135:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2442 00000100:00080000:0.0:1769076759.665139:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076759.665143:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076759.665153:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076759.665166:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961386752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076759.665177:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076759.665180:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076759.665182:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076759.665186:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2442 00000100:00080000:0.0:1769076759.665188:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076759.665191:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076759.665195:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076759.665202:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961386880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076759.665267:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961386624:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769076759.665352:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076759.665398:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076759/real 1769076759] req@ffff895e4383ce00 x1855008961386624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076759.665544:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961386624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076759.665560:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076759.665565:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076759.665570:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961386624:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076759.665580:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961386752:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076759.665638:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076759.665663:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076759/real 1769076759] req@ffff895e4383c000 x1855008961386752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076759.665678:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961386752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076759.665688:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076759.665692:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076759.665696:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961386752:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076759.665714:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961386880:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076759.665739:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076759.665781:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076759/real 1769076759] req@ffff895e4383f480 x1855008961386880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076759.665796:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961386880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076759.665805:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076759.665809:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076759.665813:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961386880:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076760.175828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076760.367856:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076761.199813:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076761.391860:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076762.223898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076762.415948:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076763.247852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076763.439863:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076764.271814:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076764.463983:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076764.783942:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076764.783954:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076764.783959:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076764.783970:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2447 00000100:00080000:1.0:1769076764.783984:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076764.783991:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076764.784037:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961387008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076764.784068:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076764.784092:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076764.784094:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076764.784099:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2447 00000100:00080000:1.0:1769076764.784103:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076764.784107:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076764.784117:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076764.784127:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961387008:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076764.784131:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961387136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076764.784144:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076764.784147:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076764.784149:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076764.784152:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2447 00000100:00080000:1.0:1769076764.784156:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076764.784159:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076764.784162:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076764.784182:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961387264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076764.784212:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076764.784266:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076764/real 1769076764] req@ffff895e4713c700 x1855008961387008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076764.784284:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961387008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076764.784297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076764.784311:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076764.784317:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961387008:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076764.784350:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961387136:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076764.784380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076764.784395:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076764/real 1769076764] req@ffff895e4713fb80 x1855008961387136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076764.784408:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961387136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076764.784457:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076764.784466:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076764.784473:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961387136:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076764.784487:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961387264:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076764.784529:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076764.784583:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076764/real 1769076764] req@ffff895e4713c000 x1855008961387264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076764.784593:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961387264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076764.784600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076764.784602:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076764.784605:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961387264:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076765.295862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076765.488019:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076766.319866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076766.511954:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076767.343915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076767.535874:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076768.367888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076768.559964:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076769.391886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076769.583928:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076769.903890:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076769.903907:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076769.903914:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076769.903928:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2452 00000100:00080000:3.0:1769076769.903944:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076769.903971:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076769.904041:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961387392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076769.904085:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076769.904091:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076769.904094:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076769.904100:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2452 00000100:00080000:3.0:1769076769.904106:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076769.904112:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076769.904126:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076769.904144:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961387520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076769.904160:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076769.904165:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769076769.904204:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961387392:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076769.904217:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076769.904227:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2452 00000100:00080000:3.0:1769076769.904232:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076769.904237:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076769.904271:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076769.904289:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961387648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076769.904358:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076769.904425:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076769/real 1769076769] req@ffff895e453d7800 x1855008961387392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076769.904442:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961387392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076769.904452:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076769.904455:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076769.904459:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961387392:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076769.904465:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961387520:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076769.904483:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076769.904490:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076769/real 1769076769] req@ffff895e453d5f80 x1855008961387520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076769.904510:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961387520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076769.904516:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076769.904518:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076769.904520:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961387520:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076769.904544:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961387648:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076769.904555:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076769.904561:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076769/real 1769076769] req@ffff895e453d4a80 x1855008961387648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076769.904594:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961387648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076769.904603:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076769.904607:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076769.904610:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961387648:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076770.415916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076770.607954:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076771.439862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076771.632044:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076772.463901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076772.655909:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076773.487847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076773.679942:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076774.511941:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076774.703897:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076775.023883:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076775.023896:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076775.023900:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076775.023910:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2457 00000100:00080000:1.0:1769076775.023920:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076775.023926:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076775.023960:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961387776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076775.023997:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076775.024000:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076775.024001:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076775.024004:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2457 00000100:00080000:1.0:1769076775.024007:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076775.024010:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076775.024017:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076775.024026:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961387904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076775.024034:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076775.024036:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076775.024037:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076775.024040:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2457 00000100:00080000:1.0:1769076775.024042:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076775.024045:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076775.024048:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076775.024054:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961387776:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076775.024057:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961388032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076775.024126:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076775.024161:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076775/real 1769076775] req@ffff895e4713ed80 x1855008961387776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076775.024175:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961387776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076775.024183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076775.024186:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076775.024189:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961387776:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076775.024195:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961387904:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076775.024211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076775.024219:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076775/real 1769076775] req@ffff895e4713d880 x1855008961387904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076775.024227:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961387904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076775.024270:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076775.024273:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076775.024275:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961387904:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076775.024296:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961388032:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076775.024313:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076775.024320:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076775/real 1769076775] req@ffff895e4713e300 x1855008961388032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076775.024328:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961388032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076775.024333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076775.024335:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076775.024337:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961388032:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076775.535952:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076775.727871:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076776.559872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076776.751959:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076777.583862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076777.775951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076778.607848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076778.799876:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076779.631909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076779.823964:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076780.144009:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076780.144031:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076780.144072:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076780.144092:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2462 00000100:00080000:2.0:1769076780.144114:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076780.144128:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076780.144226:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961388160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076780.144273:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076780.144278:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076780.144281:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076780.144293:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2462 00000100:00080000:2.0:1769076780.144299:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076780.144305:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076780.144329:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076780.144348:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961388288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076780.144368:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961388160:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076780.144368:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076780.144386:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076780.144391:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076780.144397:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2462 00000100:00080000:2.0:1769076780.144402:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076780.144418:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076780.144430:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076780.144450:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961388416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076780.144495:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076780.144568:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076780/real 1769076780] req@ffff895e44e29180 x1855008961388160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076780.144595:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961388160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076780.144610:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076780.144616:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076780.144634:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961388160:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076780.144674:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961388288:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076780.144709:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076780.144743:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076780/real 1769076780] req@ffff895e44e29500 x1855008961388288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076780.144855:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961388288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076780.144874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076780.144892:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076780.144900:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961388288:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076780.144914:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961388416:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076780.144952:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076780.144977:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076780/real 1769076780] req@ffff895e44e28a80 x1855008961388416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076780.144992:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961388416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076780.145002:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076780.145007:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076780.145011:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961388416:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076780.655901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076780.847897:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076781.679846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076781.871900:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076782.703850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076782.895879:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076783.727825:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076783.919930:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076784.751894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076784.943892:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076785.263918:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076785.263943:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076785.263949:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076785.263961:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2467 00000100:00080000:1.0:1769076785.263998:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076785.264008:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076785.264063:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961388544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076785.264151:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076785.264159:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076785.264162:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076785.264171:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2467 00000100:00080000:1.0:1769076785.264191:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076785.264201:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076785.264218:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076785.264252:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961388544:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076785.264261:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961388672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076785.264281:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076785.264286:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076785.264289:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076785.264294:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2467 00000100:00080000:1.0:1769076785.264301:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076785.264307:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076785.264324:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076785.264339:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961388800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076785.264369:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076785.264433:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076785/real 1769076785] req@ffff895e4713f100 x1855008961388544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076785.264470:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961388544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076785.264483:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076785.264489:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076785.264494:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961388544:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076785.264524:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961388672:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076785.264554:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076785.264568:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076785/real 1769076785] req@ffff895e4713ea00 x1855008961388672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076785.264595:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961388672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076785.264606:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076785.264610:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076785.264614:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961388672:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076785.264621:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961388800:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076785.264639:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076785.264649:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076785/real 1769076785] req@ffff895e4713d500 x1855008961388800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076785.264663:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961388800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076785.264702:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076785.264707:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076785.264711:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961388800:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076785.775881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076785.967997:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076786.799878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076786.991894:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076787.823891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076788.015924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076788.847851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076789.039972:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076789.871844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076790.063921:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076790.383891:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076790.383904:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076790.383908:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076790.383919:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2472 00000100:00080000:0.0:1769076790.383930:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076790.383936:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076790.383969:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961388928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076790.384014:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076790.384018:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076790.384019:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076790.384022:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2472 00000100:00080000:0.0:1769076790.384025:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076790.384028:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076790.384035:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076790.384045:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961389056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076790.384053:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076790.384055:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076790.384071:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076790.384076:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2472 00000100:00100000:3.0:1769076790.384078:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961388928:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076790.384080:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076790.384087:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076790.384092:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076790.384101:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961389184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076790.384149:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076790.384199:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076790/real 1769076790] req@ffff895e4383fb80 x1855008961388928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076790.384220:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961388928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076790.384231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076790.384236:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076790.384240:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961388928:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076790.384248:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961389056:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076790.384271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076790.384284:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076790/real 1769076790] req@ffff895e4383c380 x1855008961389056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076790.384297:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961389056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076790.384306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076790.384310:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076790.384313:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961389056:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076790.384336:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961389184:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076790.384368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076790.384380:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076790/real 1769076790] req@ffff895e4383f100 x1855008961389184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076790.384393:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961389184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076790.384401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076790.384405:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076790.384408:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961389184:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076790.895810:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076791.087868:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076791.919847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076792.111993:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076792.943880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076793.136005:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076793.967847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076794.159865:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076794.991857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076795.183921:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076795.503843:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076795.503852:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076795.503857:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076795.503868:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2478 00000100:00080000:0.0:1769076795.503879:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076795.503885:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076795.503935:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961389312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076795.503975:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076795.503978:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076795.503980:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076795.503987:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2478 00000100:00080000:0.0:1769076795.503991:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076795.503994:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076795.504004:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076795.504016:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961389440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076795.504028:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961389312:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076795.504047:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076795.504051:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076795.504053:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076795.504057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2478 00000100:00080000:0.0:1769076795.504061:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076795.504064:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076795.504069:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076795.504079:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961389568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076795.504089:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076795.504123:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076795/real 1769076795] req@ffff895e4383c700 x1855008961389312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076795.504142:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961389312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076795.504155:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076795.504160:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076795.504165:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961389312:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076795.504201:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961389440:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076795.504226:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076795.504238:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076795/real 1769076795] req@ffff895e4383e680 x1855008961389440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076795.504265:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961389440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076795.504271:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076795.504274:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076795.504276:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961389440:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076795.504281:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961389568:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076795.504294:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076795.504300:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076795/real 1769076795] req@ffff895e4383d880 x1855008961389568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076795.504308:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961389568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076795.504313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076795.504315:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076795.504317:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961389568:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076796.015840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076796.207942:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076797.039963:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076797.231893:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076798.063848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076798.255922:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076799.087878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076799.279872:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076800.111853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076800.303893:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076800.623941:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076800.623952:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076800.623957:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076800.623967:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2483 00000100:00080000:3.0:1769076800.623979:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076800.624003:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076800.624040:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961389696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076800.624092:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076800.624096:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076800.624099:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076800.624105:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2483 00000100:00080000:3.0:1769076800.624110:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076800.624118:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076800.624130:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076800.624146:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961389696:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076800.624168:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961389824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076800.624187:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076800.624191:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076800.624194:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076800.624199:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2483 00000100:00080000:3.0:1769076800.624206:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076800.624231:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076800.624240:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076800.624258:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961389952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076800.624273:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961389824:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076800.624311:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961389952:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076801.135897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076801.327930:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076802.159872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076802.351888:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076803.183954:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076803.375861:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769076803.695993:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769076803.696029:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769076803.696043:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769076803.696079:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769076803.696086:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769076803.696136:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076800/real 1769076803] req@ffff895e453d5f80 x1855008961389952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076803.696176:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961389952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076803.696190:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076803.696195:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076803.696199:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961389952:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076803.696226:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076800/real 1769076803] req@ffff895e453d4a80 x1855008961389824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076803.696237:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961389824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076803.696243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076803.696247:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076803.696249:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961389824:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076803.696254:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076800/real 1769076803] req@ffff895e453d5c00 x1855008961389696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076803.696261:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961389696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076803.696266:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076803.696268:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076803.696270:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961389696:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076804.207959:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076804.399928:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076805.231864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076805.423910:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076805.743864:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076805.743873:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076805.743876:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076805.743885:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2488 00000100:00080000:3.0:1769076805.743894:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076805.743900:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076805.743930:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961390080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076805.743953:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076805.743956:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076805.743957:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076805.743960:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2488 00000100:00080000:3.0:1769076805.743962:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076805.743965:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076805.743971:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076805.743982:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961390208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076805.744011:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961390080:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076805.744021:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076805.744039:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076805.744042:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076805.744047:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2488 00000100:00080000:3.0:1769076805.744052:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076805.744056:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076805.744062:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076805.744076:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961390336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076805.744081:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076805.744126:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076805/real 1769076805] req@ffff895e453d7800 x1855008961390080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076805.744147:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961390080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076805.744165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076805.744169:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076805.744174:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961390080:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076805.744202:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961390208:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076805.744229:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076805.744242:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076805/real 1769076805] req@ffff895e453d4e00 x1855008961390208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076805.744256:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961390208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076805.744273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076805.744277:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076805.744281:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961390208:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076805.744287:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961390336:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076805.744309:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076805.744321:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076805/real 1769076805] req@ffff895e453d7480 x1855008961390336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076805.744337:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961390336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076805.744347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076805.744351:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076805.744370:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961390336:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076806.255828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076806.447899:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076807.279836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076807.471884:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076808.303817:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076808.495954:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076809.327894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076809.519997:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076810.351864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076810.544050:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076810.864006:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076810.864059:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076810.864067:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076810.864084:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2493 00000100:00080000:1.0:1769076810.864104:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076810.864117:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076810.864183:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961390464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076810.864238:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076810.864245:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076810.864248:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076810.864255:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2493 00000100:00080000:1.0:1769076810.864261:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076810.864268:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076810.864296:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076810.864319:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961390592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076810.864338:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076810.864344:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076810.864349:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076810.864356:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2493 00000100:00080000:1.0:1769076810.864363:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076810.864372:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961390464:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769076810.864374:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076810.864397:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076810.864426:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961390720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076810.864468:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076810.864525:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076810/real 1769076810] req@ffff895e4713ce00 x1855008961390464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076810.864549:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961390464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076810.864583:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076810.864592:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076810.864600:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961390464:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076810.864615:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961390592:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076810.864725:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076810.864743:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076810/real 1769076810] req@ffff895e4713f480 x1855008961390592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076810.864801:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961390592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076810.864814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076810.864819:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076810.864840:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961390592:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076810.864872:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961390720:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076810.864901:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076810.864916:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076810/real 1769076810] req@ffff895e4713ca80 x1855008961390720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076810.864933:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961390720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076810.864943:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076810.864945:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076810.864948:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961390720:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076811.375913:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076811.567899:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076812.399911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076812.591907:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076813.423845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076813.615918:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076814.447836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076814.640007:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076815.471896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076815.663867:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076815.983884:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076815.983894:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076815.983899:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076815.983908:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2498 00000100:00080000:2.0:1769076815.983937:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076815.983944:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076815.983978:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961390848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076815.984007:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076815.984011:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076815.984013:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076815.984018:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2498 00000100:00080000:2.0:1769076815.984021:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076815.984025:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076815.984034:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076815.984049:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961390976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076815.984060:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961390848:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076815.984063:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076815.984072:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076815.984074:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076815.984079:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2498 00000100:00080000:2.0:1769076815.984083:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076815.984087:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076815.984092:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076815.984107:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961391104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076815.984133:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076815.984189:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076815/real 1769076815] req@ffff895e44e2bb80 x1855008961390848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076815.984209:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961390848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076815.984220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076815.984231:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076815.984236:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961390848:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076815.984262:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961390976:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076815.984289:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076815.984304:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076815/real 1769076815] req@ffff895e44e2b800 x1855008961390976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076815.984318:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961390976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076815.984328:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076815.984332:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076815.984336:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961390976:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076815.984342:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961391104:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076815.984364:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076815.984375:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076815/real 1769076815] req@ffff895e44e29c00 x1855008961391104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076815.984388:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961391104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076815.984397:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076815.984400:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076815.984404:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961391104:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076816.495838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076816.687898:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076817.519886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076817.711874:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076818.543895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076818.735927:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076819.567931:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076819.759931:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076820.591932:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076820.784021:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076821.103902:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076821.103911:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076821.103915:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076821.103924:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2503 00000100:00080000:0.0:1769076821.103934:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076821.103940:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076821.103974:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961391232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076821.104012:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076821.104016:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076821.104017:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076821.104021:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2503 00000100:00080000:0.0:1769076821.104023:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076821.104026:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076821.104034:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076821.104045:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961391360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076821.104055:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076821.104057:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076821.104059:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076821.104061:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2503 00000100:00080000:0.0:1769076821.104063:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076821.104069:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076821.104071:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961391232:192.168.204.117@tcp:250:kworker.0 00000008:00100000:0.0:1769076821.104092:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076821.104123:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961391488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076821.104158:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076821.104196:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076821/real 1769076821] req@ffff895e4383df80 x1855008961391232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076821.104213:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961391232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076821.104222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076821.104226:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076821.104230:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961391232:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076821.104261:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961391360:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076821.104292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076821.104320:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076821/real 1769076821] req@ffff895e4383ed80 x1855008961391360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076821.104331:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961391360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076821.104338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076821.104340:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076821.104343:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961391360:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076821.104348:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961391488:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076821.104380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076821.104394:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076821/real 1769076821] req@ffff895e4383d180 x1855008961391488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076821.104412:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961391488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076821.104441:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076821.104445:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076821.104449:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961391488:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769076821.615840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076821.807856:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076822.639901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076822.832014:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076823.663906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076823.855973:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076824.687912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076824.879979:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076825.711854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076825.903912:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076826.223958:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076826.223994:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076826.223999:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076826.224011:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2508 00000100:00080000:3.0:1769076826.224025:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076826.224034:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076826.224077:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961391616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076826.224122:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076826.224127:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076826.224129:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076826.224134:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2508 00000100:00080000:3.0:1769076826.224139:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076826.224143:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076826.224154:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076826.224171:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961391744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076826.224185:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076826.224190:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076826.224193:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076826.224197:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2508 00000100:00080000:3.0:1769076826.224201:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076826.224208:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961391616:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076826.224213:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076826.224229:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076826.224246:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961391872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076826.224340:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076826.224432:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076826/real 1769076826] req@ffff895e453d5500 x1855008961391616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076826.224465:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961391616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076826.224488:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076826.224498:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076826.224506:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961391616:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076826.224521:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961391744:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076826.224572:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076826.224593:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076826/real 1769076826] req@ffff895e453d7b80 x1855008961391744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076826.224610:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961391744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076826.224621:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076826.224626:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076826.224630:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961391744:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076826.224661:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961391872:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076826.224704:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076826.224718:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076826/real 1769076826] req@ffff895e453d7100 x1855008961391872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076826.224782:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961391872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076826.224794:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076826.224798:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076826.224803:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961391872:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076826.735902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076826.927950:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076827.759836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076827.951876:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076828.783865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076828.975872:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076829.807825:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076829.999939:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076830.831841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076831.023968:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076831.343958:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076831.343975:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076831.343982:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076831.344012:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2513 00000100:00080000:1.0:1769076831.344031:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076831.344044:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076831.344134:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961392000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076831.344197:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076831.344204:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076831.344208:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076831.344214:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2513 00000100:00080000:1.0:1769076831.344221:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076831.344228:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076831.344243:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076831.344264:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961392128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076831.344317:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961392000:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769076831.344367:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076831.344371:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076831.344373:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076831.344380:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2513 00000100:00080000:1.0:1769076831.344385:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076831.344391:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076831.344398:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076831.344411:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961392256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076831.344431:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076831.344472:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076831/real 1769076831] req@ffff895e4713f800 x1855008961392000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076831.344487:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961392000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076831.344496:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076831.344500:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076831.344503:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961392000:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076831.344528:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961392128:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076831.344542:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076831.344551:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076831/real 1769076831] req@ffff895e4713dc00 x1855008961392128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076831.344572:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961392128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076831.344578:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076831.344580:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076831.344582:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961392128:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076831.344586:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961392256:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076831.344599:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076831.344606:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076831/real 1769076831] req@ffff895e4713d180 x1855008961392256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076831.344613:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961392256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076831.344618:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076831.344620:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076831.344622:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961392256:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076831.855835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076832.047906:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076832.879898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076833.071886:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076833.903898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076834.095900:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076834.927876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076835.120044:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076835.951923:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076836.143962:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076836.463920:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076836.463933:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076836.463940:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076836.463952:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2519 00000100:00080000:3.0:1769076836.463966:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076836.463976:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076836.464022:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961392384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076836.464061:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076836.464066:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076836.464069:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076836.464074:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2519 00000100:00080000:3.0:1769076836.464078:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076836.464082:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076836.464093:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076836.464108:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961392512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076836.464123:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076836.464128:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076836.464130:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076836.464163:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2519 00000100:00100000:0.0:1769076836.464165:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961392384:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076836.464176:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076836.464189:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076836.464202:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076836.464255:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961392640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076836.464294:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076836.464362:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076836/real 1769076836] req@ffff895e453d4700 x1855008961392384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076836.464386:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961392384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076836.464419:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076836.464425:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076836.464431:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961392384:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076836.464441:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961392512:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076836.464474:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076836.464490:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076836/real 1769076836] req@ffff895e453d5880 x1855008961392512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076836.464505:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961392512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076836.464515:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076836.464520:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076836.464524:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961392512:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076836.464553:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961392640:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076836.464573:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076836.464584:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076836/real 1769076836] req@ffff895e453d5180 x1855008961392640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076836.464598:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961392640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076836.464608:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076836.464611:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076836.464615:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961392640:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076836.975896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076837.167880:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076837.999895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076838.192031:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076839.023916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076839.216010:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076840.047919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076840.239949:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076841.071904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076841.263869:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076841.583908:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076841.583919:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076841.583923:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076841.583933:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2524 00000100:00080000:1.0:1769076841.583945:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076841.583952:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076841.583991:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961392768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076841.584024:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076841.584028:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076841.584030:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076841.584034:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2524 00000100:00080000:1.0:1769076841.584037:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076841.584040:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076841.584068:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076841.584081:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961392896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076841.584092:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076841.584096:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076841.584098:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076841.584102:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2524 00000100:00080000:1.0:1769076841.584107:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076841.584112:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076841.584116:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076841.584146:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961392768:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076841.584162:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961393024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076841.584244:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076841.584289:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076841/real 1769076841] req@ffff895e4713c380 x1855008961392768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076841.584330:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961392768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076841.584344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076841.584350:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076841.584355:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961392768:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076841.584365:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961392896:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076841.584393:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076841.584407:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076841/real 1769076841] req@ffff895e4713df80 x1855008961392896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076841.584421:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961392896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076841.584432:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076841.584435:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076841.584439:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961392896:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076841.584469:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961393024:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076841.584489:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076841.584501:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076841/real 1769076841] req@ffff895e4713d180 x1855008961393024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076841.584560:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961393024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076841.584569:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076841.584572:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076841.584575:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961393024:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076842.095908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076842.288009:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076843.119901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076843.311924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076844.143840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076844.335920:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076845.167929:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076845.359978:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076846.191948:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076846.383923:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076846.703950:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076846.703960:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076846.703964:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076846.703974:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2529 00000100:00080000:0.0:1769076846.703985:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076846.703991:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076846.704031:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961393152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076846.704089:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076846.704092:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076846.704094:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076846.704102:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2529 00000100:00080000:0.0:1769076846.704105:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076846.704108:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076846.704118:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076846.704130:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961393280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076846.704152:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961393152:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076846.704174:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076846.704177:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076846.704180:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076846.704185:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2529 00000100:00080000:0.0:1769076846.704190:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076846.704194:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076846.704199:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076846.704211:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961393408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076846.704229:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076846.704281:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076846/real 1769076846] req@ffff895e4383d180 x1855008961393152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076846.704300:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961393152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076846.704311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076846.704325:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076846.704329:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961393152:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076846.704353:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961393280:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076846.704376:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076846.704388:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076846/real 1769076846] req@ffff895e4383ed80 x1855008961393280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076846.704400:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961393280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076846.704408:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076846.704411:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076846.704414:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961393280:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076846.704419:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961393408:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076846.704432:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076846.704440:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076846/real 1769076846] req@ffff895e4383df80 x1855008961393408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076846.704449:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961393408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076846.704456:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076846.704459:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076846.704461:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961393408:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076847.215843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076847.407884:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076848.239913:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076848.431973:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076849.263852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076849.455943:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076850.287835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076850.479891:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076851.311874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076851.504007:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076851.823940:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076851.823969:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076851.823975:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076851.823987:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2534 00000100:00080000:1.0:1769076851.824015:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076851.824022:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076851.824055:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961393536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076851.824079:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076851.824082:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076851.824084:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076851.824086:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2534 00000100:00080000:1.0:1769076851.824104:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076851.824108:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076851.824117:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076851.824123:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961393536:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076851.824131:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961393664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076851.824139:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076851.824142:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076851.824143:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076851.824146:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2534 00000100:00080000:1.0:1769076851.824149:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076851.824151:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076851.824154:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076851.824162:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961393792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076851.824185:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076851.824217:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076851/real 1769076851] req@ffff895e4713dc00 x1855008961393536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076851.824232:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961393536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076851.824240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076851.824243:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076851.824246:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961393536:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076851.824286:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961393664:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076851.824306:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076851.824316:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076851/real 1769076851] req@ffff895e4713f800 x1855008961393664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076851.824342:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961393664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076851.824349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076851.824352:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076851.824355:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961393664:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076851.824371:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961393792:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076851.824388:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076851.824397:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076851/real 1769076851] req@ffff895e4713ca80 x1855008961393792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076851.824406:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961393792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076851.824413:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076851.824415:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076851.824418:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961393792:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076852.335859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076852.527908:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076853.359844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076853.551978:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076854.383950:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076854.576052:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076855.408014:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076855.599929:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076856.431825:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076856.623924:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076856.943863:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076856.943913:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076856.943918:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076856.943927:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2539 00000100:00080000:0.0:1769076856.943936:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076856.943942:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076856.943973:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961393920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076856.944010:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076856.944012:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076856.944014:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076856.944017:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2539 00000100:00080000:0.0:1769076856.944019:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076856.944021:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076856.944043:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076856.944050:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961393920:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076856.944056:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961394048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076856.944066:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076856.944068:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076856.944070:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076856.944073:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2539 00000100:00080000:0.0:1769076856.944076:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076856.944078:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076856.944082:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076856.944091:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961394176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076856.944109:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076856.944148:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076856/real 1769076856] req@ffff895e4383d880 x1855008961393920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076856.944161:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961393920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076856.944169:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076856.944178:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076856.944181:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961393920:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076856.944201:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961394048:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076856.944218:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076856.944236:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076856/real 1769076856] req@ffff895e4383e680 x1855008961394048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076856.944244:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961394048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076856.944250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076856.944259:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076856.944262:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961394048:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076856.944266:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961394176:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076856.944278:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076856.944285:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076856/real 1769076856] req@ffff895e4383c700 x1855008961394176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076856.944292:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961394176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076856.944298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076856.944300:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076856.944302:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961394176:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076857.455958:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076857.647923:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076858.479904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076858.671920:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076859.503937:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076859.695987:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076860.527989:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076860.719999:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076861.551954:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076861.743921:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076862.063963:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076862.063973:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076862.063978:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076862.063988:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2544 00000100:00080000:2.0:1769076862.064000:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076862.064026:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076862.064101:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961394304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076862.064175:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076862.064181:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076862.064184:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076862.064190:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2544 00000100:00080000:2.0:1769076862.064203:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076862.064210:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076862.064220:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961394304:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769076862.064224:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076862.064250:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961394432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076862.064268:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076862.064289:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076862.064292:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076862.064300:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2544 00000800:00000100:3.0:1769076862.064304:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:2.0:1769076862.064310:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076862.064320:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076862.064328:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076862.064348:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076862/real 1769076862] req@ffff895e44e29c00 x1855008961394304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769076862.064351:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961394560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076862.064389:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961394304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076862.064401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076862.064405:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076862.064411:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961394304:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076862.064463:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961394432:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076862.064505:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076862.064515:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076862/real 1769076862] req@ffff895e44e2b800 x1855008961394432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076862.064526:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961394432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076862.064534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076862.064536:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076862.064539:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961394432:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076862.064544:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961394560:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076862.064570:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076862.064579:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076862/real 1769076862] req@ffff895e44e2bb80 x1855008961394560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076862.064589:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961394560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076862.064595:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076862.064597:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076862.064600:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961394560:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076862.575873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076862.767925:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076863.599890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076863.791954:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076864.623867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076864.815877:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076865.647884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076865.839941:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076866.671906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076866.864011:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076867.183930:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076867.183946:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076867.183954:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076867.183970:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2549 00000100:00080000:0.0:1769076867.183990:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076867.184001:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076867.184065:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961394688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076867.184114:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076867.184119:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076867.184122:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076867.184127:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2549 00000100:00080000:0.0:1769076867.184156:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076867.184162:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076867.184186:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076867.184205:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961394816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076867.184221:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076867.184226:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076867.184229:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076867.184235:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2549 00000100:00080000:0.0:1769076867.184240:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076867.184245:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076867.184252:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076867.184271:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961394688:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076867.184284:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961394944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076867.184485:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961394816:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076867.184550:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961394944:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076867.695904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076867.887920:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076868.719821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076868.911872:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076869.743915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076869.936028:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769076870.256082:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769076870.256130:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076870.256164:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769076870.256195:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769076870.256202:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769076870.256236:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076867/real 1769076870] req@ffff895e4383fb80 x1855008961394944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076870.256267:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961394944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076870.256276:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076870.256280:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076870.256283:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961394944:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076870.256289:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076867/real 1769076870] req@ffff895e4383f100 x1855008961394688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076870.256313:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961394688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076870.256319:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076870.256321:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076870.256324:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961394688:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076870.256328:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076867/real 1769076870] req@ffff895e4383c380 x1855008961394816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076870.256335:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961394816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076870.256340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076870.256342:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076870.256344:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961394816:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076870.767921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769076870.959964:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076871.791914:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076871.983967:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076872.303953:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076872.303967:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076872.303972:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076872.304003:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2554 00000100:00080000:2.0:1769076872.304033:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076872.304042:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076872.304082:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961395072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076872.304134:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076872.304141:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076872.304145:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076872.304154:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2554 00000100:00080000:2.0:1769076872.304162:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076872.304170:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076872.304188:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076872.304192:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961395072:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076872.304221:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961395200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076872.304242:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076872.304246:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076872.304250:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076872.304254:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2554 00000100:00080000:2.0:1769076872.304271:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076872.304284:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:3.0:1769076872.304284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:2.0:1769076872.304301:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076872.304330:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961395328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076872.304360:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076872/real 1769076872] req@ffff895e44e28a80 x1855008961395072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076872.304393:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961395072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076872.304415:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076872.304422:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076872.304450:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961395072:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076872.304492:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961395200:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076872.304536:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076872.304560:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076872/real 1769076872] req@ffff895e44e29500 x1855008961395200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076872.304584:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961395200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076872.304669:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076872.304677:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076872.304683:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961395200:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076872.304696:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961395328:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076872.304738:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076872.304828:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076872/real 1769076872] req@ffff895e44e29180 x1855008961395328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076872.304856:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961395328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076872.304870:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076872.304876:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076872.304881:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961395328:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076872.815969:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076873.007964:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076873.839874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076874.031906:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076874.863863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076875.056001:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076875.887862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076876.080011:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076876.911891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076877.104010:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076877.423849:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076877.423859:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076877.423863:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076877.423872:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2559 00000100:00080000:2.0:1769076877.423884:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076877.423893:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076877.423936:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961395456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076877.423971:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076877.423975:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076877.423976:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076877.423980:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2559 00000100:00080000:2.0:1769076877.423984:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076877.423988:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076877.423998:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076877.424012:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961395584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076877.424040:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961395456:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076877.424063:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076877.424066:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076877.424068:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076877.424073:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2559 00000100:00080000:2.0:1769076877.424078:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076877.424082:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076877.424088:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076877.424100:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961395712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076877.424122:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076877.424181:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076877/real 1769076877] req@ffff895e44e28e00 x1855008961395456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076877.424207:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961395456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076877.424222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076877.424228:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076877.424234:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961395456:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076877.424265:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961395584:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076877.424296:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076877.424314:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076877/real 1769076877] req@ffff895e44e2b480 x1855008961395584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076877.424333:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961395584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076877.424345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076877.424350:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076877.424355:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961395584:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076877.424365:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961395712:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076877.424391:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076877.424406:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076877/real 1769076877] req@ffff895e44e29f80 x1855008961395712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076877.424422:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961395712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076877.424451:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076877.424457:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076877.424462:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961395712:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076877.935892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076878.127910:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076878.959898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076879.152047:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076879.983967:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076880.175952:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076881.007882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076881.200031:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076882.031896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076882.223983:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076882.543865:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076882.543876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076882.543880:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076882.543888:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2565 00000100:00080000:2.0:1769076882.543898:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076882.543904:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076882.543935:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961395840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076882.543957:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076882.543973:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076882.543975:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076882.543978:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2565 00000100:00080000:2.0:1769076882.543982:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076882.543986:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076882.543994:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076882.544002:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961395840:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076882.544006:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961395968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076882.544016:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076882.544018:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076882.544020:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076882.544023:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2565 00000100:00080000:2.0:1769076882.544025:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076882.544027:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076882.544032:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076882.544045:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961396096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076882.544069:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076882.544133:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076882/real 1769076882] req@ffff895e44e2b100 x1855008961395840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076882.544165:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961395840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076882.544184:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076882.544191:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076882.544199:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961395840:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076882.544246:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961395968:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076882.544265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076882.544274:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076882/real 1769076882] req@ffff895e44e2a300 x1855008961395968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076882.544282:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961395968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076882.544299:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076882.544318:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076882.544323:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961395968:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076882.544330:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961396096:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076882.544354:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076882.544366:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076882/real 1769076882] req@ffff895e44e2aa00 x1855008961396096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076882.544380:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961396096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076882.544389:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076882.544393:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076882.544396:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961396096:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076883.055972:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076883.248087:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076884.080150:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076884.272025:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076885.103955:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076885.296017:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076886.127907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076886.320011:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076887.151896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076887.344018:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076887.663957:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076887.664019:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076887.664023:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076887.664034:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2570 00000100:00080000:1.0:1769076887.664046:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076887.664054:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076887.664090:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961396224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076887.664122:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076887.664126:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076887.664128:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076887.664131:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2570 00000100:00080000:1.0:1769076887.664135:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076887.664138:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076887.664147:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076887.664157:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961396352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076887.664168:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076887.664172:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076887.664174:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076887.664177:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2570 00000100:00080000:1.0:1769076887.664180:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076887.664184:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076887.664188:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076887.664197:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961396480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076887.664241:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961396224:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769076887.664380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076887.664445:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076887/real 1769076887] req@ffff895e4713f480 x1855008961396224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076887.664467:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961396224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076887.664505:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076887.664510:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076887.664515:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961396224:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076887.664523:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961396352:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076887.664551:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076887.664581:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076887/real 1769076887] req@ffff895e4713ce00 x1855008961396352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076887.664595:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961396352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076887.664604:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076887.664608:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076887.664611:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961396352:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076887.664639:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961396480:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076887.664659:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076887.664668:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076887/real 1769076887] req@ffff895e4713d500 x1855008961396480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076887.664680:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961396480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076887.664705:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076887.664708:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076887.664711:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961396480:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076888.176058:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076888.367988:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076889.200049:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076889.391886:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076890.223978:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076890.416067:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076891.247902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076891.440010:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076892.271956:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076892.464034:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076892.783906:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076892.783923:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076892.783954:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076892.783969:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2575 00000100:00080000:2.0:1769076892.783982:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076892.783991:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076892.784030:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961396608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076892.784067:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076892.784071:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076892.784074:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076892.784078:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2575 00000100:00080000:2.0:1769076892.784082:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076892.784086:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076892.784097:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076892.784110:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961396736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076892.784124:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076892.784129:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076892.784131:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076892.784136:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2575 00000100:00080000:2.0:1769076892.784139:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076892.784146:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076892.784152:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076892.784164:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961396608:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076892.784166:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961396864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076892.784284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076892.784349:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076892/real 1769076892] req@ffff895e44e28000 x1855008961396608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076892.784383:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961396608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076892.784401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076892.784408:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076892.784416:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961396608:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076892.784430:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961396736:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076892.784468:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076892.784486:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076892/real 1769076892] req@ffff895e44e28380 x1855008961396736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076892.784519:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961396736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076892.784529:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076892.784533:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076892.784537:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961396736:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076892.784583:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961396864:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076892.784610:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076892.784623:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076892/real 1769076892] req@ffff895e44e29880 x1855008961396864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076892.784637:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961396864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076892.784646:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076892.784650:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076892.784654:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961396864:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076893.295907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076893.487982:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076894.319805:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076894.511933:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076895.343832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076895.535884:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076896.367981:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076896.559919:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076897.391854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076897.583944:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076897.903941:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076897.903954:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076897.903958:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076897.903969:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2580 00000100:00080000:1.0:1769076897.903983:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076897.903991:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076897.904032:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961396992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076897.904059:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076897.904061:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076897.904063:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076897.904066:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2580 00000100:00080000:1.0:1769076897.904069:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076897.904073:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076897.904081:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076897.904092:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961397120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076897.904101:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769076897.904107:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961396992:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769076897.904119:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076897.904122:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076897.904126:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2580 00000100:00080000:1.0:1769076897.904130:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076897.904133:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076897.904138:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076897.904150:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961397248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076897.904197:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076897.904248:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076897/real 1769076897] req@ffff895e4713ea00 x1855008961396992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076897.904264:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961396992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076897.904273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076897.904276:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076897.904280:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961396992:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076897.904301:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961397120:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076897.904339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076897.904348:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076897/real 1769076897] req@ffff895e4713f100 x1855008961397120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076897.904358:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961397120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076897.904364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076897.904367:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076897.904369:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961397120:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076897.904374:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961397248:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076897.904385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076897.904391:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076897/real 1769076897] req@ffff895e4713e300 x1855008961397248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076897.904399:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961397248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076897.904405:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076897.904407:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076897.904409:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961397248:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076898.415836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076898.607901:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076899.439830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076899.631913:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076900.463996:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076900.656008:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076901.487901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076901.679964:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076902.511854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076902.704075:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076903.023938:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076903.023948:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076903.023952:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076903.023971:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2585 00000100:00080000:0.0:1769076903.024002:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076903.024010:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076903.024047:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961397376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076903.024080:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076903.024083:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076903.024084:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076903.024088:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2585 00000100:00080000:0.0:1769076903.024102:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076903.024108:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076903.024115:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076903.024125:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961397504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076903.024150:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076903.024152:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076903.024154:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076903.024157:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2585 00000100:00080000:0.0:1769076903.024162:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076903.024165:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076903.024169:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076903.024176:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961397632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076903.024198:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961397376:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769076903.024268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076903.024300:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076903/real 1769076903] req@ffff895e4383f480 x1855008961397376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076903.024340:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961397376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076903.024348:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076903.024352:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076903.024355:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961397376:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076903.024361:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961397504:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076903.024379:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076903.024389:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076903/real 1769076903] req@ffff895e4383c000 x1855008961397504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076903.024398:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961397504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076903.024404:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076903.024406:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076903.024409:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961397504:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076903.024427:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961397632:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076903.024438:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076903.024453:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076903/real 1769076903] req@ffff895e4383ce00 x1855008961397632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076903.024461:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961397632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076903.024477:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076903.024479:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076903.024482:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961397632:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076903.535932:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076903.727938:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076904.559915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076904.751961:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076905.583816:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076905.776028:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076906.607893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076906.800212:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076907.631946:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076907.823986:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076908.143859:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076908.143872:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076908.143877:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076908.143889:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2590 00000100:00080000:0.0:1769076908.143902:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076908.143911:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076908.143959:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961397760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076908.143991:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076908.143995:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076908.143996:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076908.144000:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2590 00000100:00080000:0.0:1769076908.144003:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076908.144020:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076908.144031:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076908.144045:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961397888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076908.144060:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076908.144066:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769076908.144066:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961397760:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076908.144073:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076908.144081:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2590 00000100:00080000:0.0:1769076908.144086:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076908.144092:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076908.144099:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076908.144141:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961398016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076908.144162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076908.144211:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076908/real 1769076908] req@ffff895e4383ca80 x1855008961397760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076908.144230:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961397760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076908.144240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076908.144244:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076908.144248:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961397760:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076908.144276:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961397888:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076908.144295:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076908.144306:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076908/real 1769076908] req@ffff895e4383ea00 x1855008961397888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076908.144315:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961397888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076908.144344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076908.144348:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076908.144353:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961397888:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076908.144362:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961398016:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076908.144390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076908.144402:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076908/real 1769076908] req@ffff895e4383dc00 x1855008961398016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076908.144412:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961398016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076908.144418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076908.144421:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076908.144424:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961398016:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076908.655846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076908.847961:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076909.679847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076909.871976:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076910.703929:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076910.895949:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076911.727937:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076911.919918:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076912.751859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076912.944052:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076913.263878:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076913.263891:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076913.263896:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076913.263908:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2595 00000100:00080000:0.0:1769076913.263943:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076913.263953:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076913.263998:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961398144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076913.264037:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076913.264044:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076913.264047:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076913.264054:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2595 00000100:00080000:0.0:1769076913.264069:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076913.264076:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076913.264093:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076913.264116:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961398272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076913.264129:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961398144:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076913.264136:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076913.264142:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076913.264145:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076913.264174:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2595 00000100:00080000:0.0:1769076913.264180:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076913.264187:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076913.264196:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076913.264213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076913.264214:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961398400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076913.264269:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076913/real 1769076913] req@ffff895e4383f800 x1855008961398144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076913.264300:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961398144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076913.264316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076913.264321:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076913.264339:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961398144:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076913.264368:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961398272:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076913.264400:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076913.264416:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076913/real 1769076913] req@ffff895e4383e300 x1855008961398272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076913.264430:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961398272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076913.264440:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076913.264444:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076913.264448:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961398272:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076913.264454:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961398400:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076913.264472:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076913.264482:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076913/real 1769076913] req@ffff895e4383d500 x1855008961398400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076913.264494:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961398400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076913.264503:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076913.264506:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076913.264510:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961398400:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076913.776009:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076913.967896:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076914.799855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076914.991956:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076915.823895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076916.015952:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076916.847947:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076917.039930:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076917.871889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076918.063897:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076918.383897:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076918.383916:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076918.383920:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076918.383929:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2600 00000100:00080000:0.0:1769076918.383939:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076918.383946:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076918.383984:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961398528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076918.384017:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076918.384020:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076918.384022:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076918.384026:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2600 00000100:00080000:0.0:1769076918.384028:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076918.384031:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076918.384039:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076918.384052:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961398656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076918.384071:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961398528:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076918.384080:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076918.384090:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076918.384092:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076918.384096:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2600 00000100:00080000:0.0:1769076918.384100:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076918.384103:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076918.384108:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076918.384120:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961398784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076918.384173:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076918.384219:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076918/real 1769076918] req@ffff895e4383d500 x1855008961398528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076918.384241:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961398528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076918.384253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076918.384259:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076918.384264:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961398528:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076918.384289:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961398656:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076918.384316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076918.384332:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076918/real 1769076918] req@ffff895e4383e300 x1855008961398656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076918.384349:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961398656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076918.384360:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076918.384382:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076918.384387:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961398656:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076918.384396:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961398784:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076918.384423:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076918.384438:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076918/real 1769076918] req@ffff895e4383f800 x1855008961398784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076918.384477:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961398784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076918.384489:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076918.384494:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076918.384499:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961398784:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076918.895986:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076919.088059:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076919.919921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076920.112028:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076920.943870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076921.136007:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076921.967851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076922.159930:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076922.991846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076923.183930:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076923.503869:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076923.503881:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076923.503886:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076923.503897:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2606 00000100:00080000:3.0:1769076923.503910:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076923.503917:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076923.503980:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961398912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076923.504060:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076923.504065:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076923.504067:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076923.504073:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2606 00000100:00080000:3.0:1769076923.504078:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076923.504084:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076923.504095:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076923.504112:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961399040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076923.504123:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961398912:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076923.504153:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076923.504158:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076923.504161:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076923.504183:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2606 00000100:00080000:3.0:1769076923.504188:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076923.504193:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076923.504201:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076923.504208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076923.504218:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961399168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076923.504248:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076923/real 1769076923] req@ffff895e453d6680 x1855008961398912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076923.504271:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961398912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076923.504309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076923.504314:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076923.504319:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961398912:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076923.504347:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961399040:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076923.504390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076923.504415:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076923/real 1769076923] req@ffff895e453d4380 x1855008961399040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076923.504431:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961399040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076923.504441:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076923.504445:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076923.504449:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961399040:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076923.504468:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961399168:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076923.504493:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076923.504542:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076923/real 1769076923] req@ffff895e453d6d80 x1855008961399168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076923.504552:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961399168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076923.504559:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076923.504561:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076923.504564:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961399168:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076924.015848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076924.207894:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076925.039881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076925.231931:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076926.063852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076926.256071:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076927.087956:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076927.280052:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076928.111855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076928.303953:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076928.623919:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076928.623935:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076928.623941:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076928.623956:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2611 00000100:00080000:1.0:1769076928.623973:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076928.623984:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076928.624045:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961399296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076928.624089:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076928.624094:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076928.624096:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076928.624101:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2611 00000100:00080000:1.0:1769076928.624105:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076928.624110:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076928.624123:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076928.624139:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961399424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076928.624154:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076928.624158:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076928.624161:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076928.624166:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2611 00000100:00080000:1.0:1769076928.624170:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076928.624177:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076928.624186:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961399296:192.168.204.117@tcp:250:kworker.0 00000008:00100000:1.0:1769076928.624210:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076928.624233:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961399552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076928.624300:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076928.624372:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076928/real 1769076928] req@ffff895e4713d880 x1855008961399296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076928.624414:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961399296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076928.624428:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076928.624434:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076928.624440:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961399296:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076928.624451:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961399424:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076928.624492:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076928.624508:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076928/real 1769076928] req@ffff895e4713ed80 x1855008961399424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076928.624523:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961399424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076928.624534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076928.624539:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076928.624543:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961399424:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076928.624578:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961399552:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076928.624598:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076928.624610:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076928/real 1769076928] req@ffff895e4713c000 x1855008961399552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076928.624624:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961399552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076928.624651:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076928.624655:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076928.624659:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961399552:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076929.135839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076929.327981:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076930.159887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076930.351905:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076931.183907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076931.376018:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076932.207856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076932.399859:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076933.231887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076933.423921:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076933.743903:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076933.743943:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076933.743949:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076933.743960:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2616 00000100:00080000:0.0:1769076933.743973:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076933.743983:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076933.744041:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961399680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076933.744080:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076933.744084:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076933.744089:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076933.744093:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2616 00000100:00080000:0.0:1769076933.744098:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076933.744103:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076933.744114:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076933.744130:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961399808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076933.744137:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961399680:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076933.744149:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076933.744170:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076933.744173:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076933.744178:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2616 00000100:00080000:0.0:1769076933.744184:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076933.744188:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076933.744194:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076933.744208:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961399936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076933.744276:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961399808:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076933.744326:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961399936:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076934.255834:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076934.447958:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076935.279888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076935.472060:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076936.303903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076936.495917:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769076936.816198:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:0.0:1769076936.816259:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769076936.816282:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:0.0:1769076936.816342:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:0.0:1769076936.816355:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769076936.816416:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076933/real 1769076936] req@ffff895e4383ea00 x1855008961399808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076936.816456:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961399808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076936.816473:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076936.816481:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076936.816487:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961399808:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076936.816500:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076933/real 1769076936] req@ffff895e4383ca80 x1855008961399936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076936.816568:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961399936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076936.816580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076936.816586:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076936.816591:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961399936:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076936.816722:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076933/real 1769076936] req@ffff895e4383dc00 x1855008961399680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076936.816783:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961399680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076936.816796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076936.816801:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076936.816807:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961399680:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076937.327915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076937.519951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076938.351798:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076938.543914:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076938.864054:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076938.864068:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076938.864073:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076938.864085:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2621 00000100:00080000:1.0:1769076938.864101:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076938.864109:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076938.864169:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961400064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076938.864198:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076938.864202:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076938.864204:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076938.864208:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2621 00000100:00080000:1.0:1769076938.864211:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076938.864215:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076938.864227:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076938.864258:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961400192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076938.864281:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961400064:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769076938.864313:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076938.864321:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076938.864326:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076938.864337:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2621 00000100:00080000:1.0:1769076938.864346:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076938.864355:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076938.864368:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076938.864415:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961400320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076938.864432:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076938.864519:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076938/real 1769076938] req@ffff895e4713fb80 x1855008961400064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076938.864548:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961400064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076938.864566:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076938.864573:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076938.864579:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961400064:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076938.864633:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961400192:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076938.864672:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076938.864693:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076938/real 1769076938] req@ffff895e4713c700 x1855008961400192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076938.864712:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961400192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076938.864725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076938.864730:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076938.864735:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961400192:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076938.864882:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961400320:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076938.864918:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076938.864936:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076938/real 1769076938] req@ffff895e4713c000 x1855008961400320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076938.864953:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961400320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076938.864964:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076938.864969:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076938.864975:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961400320:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076939.375914:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076939.567951:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076940.399861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076940.591968:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076941.423894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076941.615951:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076942.447845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076942.639920:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076943.471884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076943.664077:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076943.983976:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076943.983992:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076943.983999:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076943.984013:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2626 00000100:00080000:2.0:1769076943.984032:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076943.984043:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076943.984106:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961400448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076943.984155:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076943.984160:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076943.984174:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076943.984179:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2626 00000100:00080000:2.0:1769076943.984184:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076943.984189:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076943.984204:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076943.984222:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961400576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076943.984238:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076943.984243:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076943.984249:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769076943.984252:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961400448:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769076943.984257:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2626 00000100:00080000:2.0:1769076943.984303:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076943.984310:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076943.984318:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769076943.984336:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961400704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076943.984431:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076943.984506:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076943/real 1769076943] req@ffff895e44e29880 x1855008961400448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076943.984536:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961400448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076943.984551:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076943.984558:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076943.984564:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961400448:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076943.984601:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961400576:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076943.984633:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076943.984649:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076943/real 1769076943] req@ffff895e44e28380 x1855008961400576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076943.984665:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961400576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076943.984676:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076943.984680:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076943.984685:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961400576:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076943.984692:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961400704:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076943.984711:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076943.984727:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076943/real 1769076943] req@ffff895e44e28000 x1855008961400704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076943.984837:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961400704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076943.984853:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076943.984858:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076943.984862:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961400704:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076944.495882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076944.688024:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076945.519901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076945.711979:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076946.544036:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076946.735993:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076947.567891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076947.759953:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076948.591856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076948.783992:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076949.103946:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076949.103959:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076949.103964:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076949.103975:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2631 00000100:00080000:3.0:1769076949.103989:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076949.103997:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076949.104059:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961400832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076949.104115:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076949.104121:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076949.104124:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076949.104131:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2631 00000100:00080000:3.0:1769076949.104138:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076949.104145:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076949.104160:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076949.104175:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961400832:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076949.104230:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961400960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076949.104263:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076949.104267:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076949.104269:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076949.104273:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2631 00000100:00080000:3.0:1769076949.104290:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076949.104295:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076949.104302:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076949.104304:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076949.104347:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961401088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076949.104432:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076949/real 1769076949] req@ffff895e453d6300 x1855008961400832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076949.104458:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961400832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076949.104471:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076949.104477:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076949.104501:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961400832:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076949.104538:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961400960:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076949.104604:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076949.104621:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076949/real 1769076949] req@ffff895e453d6d80 x1855008961400960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076949.104638:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961400960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076949.104650:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076949.104655:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076949.104659:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961400960:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076949.104668:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961401088:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076949.104706:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076949.104721:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076949/real 1769076949] req@ffff895e453d4380 x1855008961401088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076949.104735:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961401088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076949.104776:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076949.104781:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076949.104786:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961401088:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076949.615869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076949.808105:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076950.639871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076950.831978:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076951.663867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076951.855953:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076952.687909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076952.880016:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076953.711906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076953.904086:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076954.223958:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076954.223987:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076954.223994:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076954.224010:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2636 00000100:00080000:1.0:1769076954.224028:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076954.224039:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076954.224112:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961401216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076954.224209:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076954.224217:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076954.224221:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076954.224231:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2636 00000100:00080000:1.0:1769076954.224306:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076954.224317:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076954.224349:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076954.224374:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961401216:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076954.224379:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961401344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076954.224410:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076954.224417:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076954.224421:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076954.224430:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2636 00000100:00080000:1.0:1769076954.224447:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076954.224452:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076954.224461:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076954.224480:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961401472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076954.224530:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076954.224583:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076954/real 1769076954] req@ffff895e4713ed80 x1855008961401216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076954.224644:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961401216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076954.224657:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076954.224663:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076954.224667:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961401216:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076954.224701:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961401344:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076954.224728:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076954.224741:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076954/real 1769076954] req@ffff895e4713d880 x1855008961401344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076954.224755:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961401344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076954.224765:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076954.224769:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076954.224772:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961401344:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076954.224779:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961401472:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076954.224812:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076954.224898:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076954/real 1769076954] req@ffff895e4713e300 x1855008961401472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076954.224914:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961401472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076954.224937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076954.224941:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076954.224945:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961401472:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076954.735880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076954.927911:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076955.759898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076955.951951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076956.783916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076956.975991:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076957.807842:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076958.000039:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769076958.831888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076959.023908:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076959.343819:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076959.343827:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076959.343832:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076959.343840:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2641 00000100:00080000:3.0:1769076959.343849:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076959.343861:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076959.343916:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961401600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076959.343947:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076959.343952:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076959.343954:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076959.343958:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2641 00000100:00080000:3.0:1769076959.343962:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076959.343986:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076959.343995:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076959.344007:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961401600:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076959.344009:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961401728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076959.344023:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076959.344026:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076959.344028:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076959.344032:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2641 00000100:00080000:3.0:1769076959.344035:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076959.344038:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076959.344042:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076959.344058:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961401856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076959.344070:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076959.344104:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076959/real 1769076959] req@ffff895e453d6680 x1855008961401600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076959.344117:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961401600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076959.344125:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076959.344128:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076959.344131:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961401600:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076959.344179:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961401728:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076959.344212:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076959.344222:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076959/real 1769076959] req@ffff895e453d5180 x1855008961401728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076959.344231:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961401728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076959.344243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076959.344245:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076959.344248:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961401728:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076959.344252:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961401856:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076959.344265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076959.344272:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076959/real 1769076959] req@ffff895e453d5880 x1855008961401856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076959.344279:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961401856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076959.344284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076959.344286:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076959.344288:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961401856:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076959.855923:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076960.047960:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076960.879859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076961.071942:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076961.903800:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076962.095975:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076962.927808:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076963.120003:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076963.951842:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076964.144119:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076964.463931:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076964.463944:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076964.463949:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076964.463960:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2647 00000100:00080000:0.0:1769076964.463974:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076964.463983:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076964.464023:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961401984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076964.464079:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076964.464111:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076964.464114:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076964.464120:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2647 00000100:00080000:0.0:1769076964.464127:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769076964.464131:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961401984:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076964.464137:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076964.464153:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076964.464174:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961402112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076964.464191:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076964.464195:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076964.464198:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076964.464204:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2647 00000100:00080000:0.0:1769076964.464209:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076964.464217:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:3.0:1769076964.464219:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:0.0:1769076964.464230:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076964.464248:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961402240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076964.464277:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076964/real 1769076964] req@ffff895e4383ce00 x1855008961401984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076964.464303:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961401984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076964.464317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076964.464322:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076964.464328:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961401984:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076964.464363:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961402112:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076964.464387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076964.464399:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076964/real 1769076964] req@ffff895e4383c000 x1855008961402112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076964.464410:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961402112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076964.464417:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076964.464420:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076964.464423:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961402112:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076964.464428:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961402240:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076964.464440:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076964.464448:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076964/real 1769076964] req@ffff895e4383f480 x1855008961402240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076964.464456:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961402240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076964.464463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076964.464465:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076964.464467:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961402240:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076964.975877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076965.167934:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076965.999888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076966.191996:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076967.023850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076967.215915:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076968.047906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076968.239955:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076969.071881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076969.263849:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076969.583953:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076969.583970:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076969.584015:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076969.584051:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2652 00000100:00080000:3.0:1769076969.584065:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076969.584073:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076969.584114:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961402368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076969.584168:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076969.584172:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076969.584175:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076969.584180:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2652 00000100:00080000:3.0:1769076969.584184:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076969.584188:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076969.584199:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076969.584215:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961402496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076969.584228:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076969.584233:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076969.584237:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769076969.584239:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961402368:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076969.584244:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2652 00000100:00080000:3.0:1769076969.584252:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076969.584259:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076969.584264:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076969.584279:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961402624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076969.584372:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076969.584437:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076969/real 1769076969] req@ffff895e453d4700 x1855008961402368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076969.584465:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961402368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076969.584489:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076969.584495:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076969.584501:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961402368:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076969.584531:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961402496:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076969.584567:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076969.584583:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076969/real 1769076969] req@ffff895e453d7100 x1855008961402496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076969.584599:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961402496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076969.584610:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076969.584614:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076969.584619:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961402496:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076969.584626:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961402624:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076969.584646:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076969.584673:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076969/real 1769076969] req@ffff895e453d7b80 x1855008961402624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076969.584693:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961402624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076969.584709:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076969.584716:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076969.584722:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961402624:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076970.095891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076970.288084:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076971.119840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769076971.311872:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076972.143851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076972.336020:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076973.167876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076973.359882:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076974.191968:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076974.383962:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076974.703886:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076974.703907:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076974.703912:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076974.703934:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2657 00000100:00080000:0.0:1769076974.703947:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076974.703954:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076974.703996:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961402752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076974.704027:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076974.704030:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076974.704031:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076974.704035:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2657 00000100:00080000:0.0:1769076974.704037:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076974.704041:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076974.704057:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076974.704069:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961402880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076974.704079:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076974.704083:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769076974.704083:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961402752:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076974.704085:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076974.704098:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2657 00000100:00080000:0.0:1769076974.704101:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076974.704123:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076974.704129:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076974.704154:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961403008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076974.704196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076974.704270:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076974/real 1769076974] req@ffff895e4383c380 x1855008961402752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076974.704308:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961402752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076974.704317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076974.704321:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076974.704324:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961402752:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076974.704344:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961402880:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076974.704362:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076974.704387:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076974/real 1769076974] req@ffff895e4383f100 x1855008961402880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076974.704395:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961402880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076974.704401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076974.704403:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076974.704405:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961402880:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076974.704409:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961403008:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076974.704421:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076974.704442:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076974/real 1769076974] req@ffff895e4383fb80 x1855008961403008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076974.704451:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961403008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076974.704456:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076974.704458:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076974.704460:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961403008:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076975.215877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076975.407896:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076976.239970:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076976.431988:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076977.263979:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076977.455998:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076978.287956:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076978.479963:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076979.311949:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076979.503942:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769076979.823900:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076979.823915:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769076979.823921:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076979.823933:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2662 00000100:00080000:2.0:1769076979.823950:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076979.823960:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769076979.824024:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961403136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076979.824071:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076979.824076:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769076979.824079:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076979.824084:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2662 00000100:00080000:2.0:1769076979.824089:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076979.824134:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076979.824149:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076979.824157:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961403136:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769076979.824168:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961403264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769076979.824184:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769076979.824189:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769076979.824191:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769076979.824197:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2662 00000100:00080000:2.0:1769076979.824204:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769076979.824210:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769076979.824219:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076979.824246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769076979.824253:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961403392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076979.824289:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076979/real 1769076979] req@ffff895e44e2aa00 x1855008961403136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076979.824318:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961403136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076979.824328:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076979.824331:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076979.824334:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961403136:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076979.824355:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961403264:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076979.824370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076979.824378:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076979/real 1769076979] req@ffff895e44e2a300 x1855008961403264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076979.824395:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961403264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076979.824415:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076979.824418:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076979.824420:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961403264:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076979.824425:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961403392:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076979.824440:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076979.824447:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076979/real 1769076979] req@ffff895e44e2b100 x1855008961403392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076979.824455:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961403392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076979.824460:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076979.824462:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076979.824463:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961403392:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076980.335911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076980.527914:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076981.359828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076981.551912:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076982.383834:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076982.575942:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076983.407999:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076983.599998:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769076984.431861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076984.623929:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076984.943910:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076984.943922:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076984.943928:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076984.943939:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2667 00000100:00080000:0.0:1769076984.943953:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076984.943960:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076984.944008:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961403520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076984.944038:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076984.944041:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076984.944042:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076984.944045:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2667 00000100:00080000:0.0:1769076984.944048:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076984.944050:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076984.944059:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076984.944069:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961403648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076984.944095:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961403520:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076984.944110:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076984.944113:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076984.944115:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076984.944119:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2667 00000100:00080000:0.0:1769076984.944123:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076984.944126:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076984.944130:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076984.944140:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961403776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076984.944177:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076984.944227:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076984/real 1769076984] req@ffff895e4383c700 x1855008961403520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076984.944243:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961403520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076984.944263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076984.944267:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076984.944270:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961403520:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076984.944294:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961403648:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076984.944312:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076984.944321:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076984/real 1769076984] req@ffff895e4383e680 x1855008961403648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076984.944330:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961403648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076984.944336:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076984.944338:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076984.944341:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961403648:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076984.944344:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961403776:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076984.944354:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076984.944360:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076984/real 1769076984] req@ffff895e4383d880 x1855008961403776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076984.944367:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961403776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076984.944372:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076984.944374:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076984.944376:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961403776:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769076985.455871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076985.647911:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076986.479885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076986.671871:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076987.503868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076987.695967:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076988.527843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076988.719972:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076989.552030:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076989.743995:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076990.063964:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076990.063983:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076990.063990:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076990.064005:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2672 00000100:00080000:1.0:1769076990.064022:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076990.064035:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076990.064088:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961403904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076990.064132:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076990.064140:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076990.064144:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076990.064152:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2672 00000100:00080000:1.0:1769076990.064160:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076990.064169:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076990.064184:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076990.064242:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961404032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076990.064244:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961403904:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769076990.064274:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076990.064278:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076990.064282:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076990.064289:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2672 00000100:00080000:1.0:1769076990.064300:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076990.064309:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076990.064321:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076990.064351:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961404160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076990.064354:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076990.064434:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076990/real 1769076990] req@ffff895e4713f100 x1855008961403904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076990.064470:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961403904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076990.064484:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076990.064490:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076990.064496:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961403904:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076990.064527:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961404032:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076990.064560:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076990.064596:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076990/real 1769076990] req@ffff895e4713ea00 x1855008961404032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076990.064613:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961404032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076990.064698:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076990.064705:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076990.064711:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961404032:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076990.064740:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961404160:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076990.064836:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076990.064857:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076990/real 1769076990] req@ffff895e4713d500 x1855008961404160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076990.064878:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961404160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076990.064900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076990.064907:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076990.064912:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961404160:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076990.575882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076990.767967:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076991.599889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076991.792022:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076992.623922:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076992.815919:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076993.647878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076993.840050:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769076994.671915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076994.864036:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076995.183945:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076995.183963:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076995.183970:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076995.183983:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2677 00000100:00080000:3.0:1769076995.183999:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076995.184012:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076995.184069:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961404288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076995.184128:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076995.184136:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076995.184163:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076995.184172:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2677 00000100:00080000:3.0:1769076995.184182:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076995.184189:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076995.184194:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961404288:192.168.204.117@tcp:250:kworker.0 00000008:00100000:3.0:1769076995.184205:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076995.184228:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961404416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076995.184250:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076995.184257:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076995.184260:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076995.184267:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2677 00000100:00080000:3.0:1769076995.184272:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076995.184278:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076995.184287:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769076995.184302:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076995.184304:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961404544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076995.184387:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076995/real 1769076995] req@ffff895e453d5500 x1855008961404288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076995.184422:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961404288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076995.184442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076995.184450:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076995.184467:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961404288:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076995.184507:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961404416:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076995.184603:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076995.184624:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076995/real 1769076995] req@ffff895e453d7480 x1855008961404416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076995.184641:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961404416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076995.184652:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076995.184656:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076995.184660:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961404416:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076995.184669:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961404544:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076995.184691:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076995.184703:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076995/real 1769076995] req@ffff895e453d4e00 x1855008961404544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076995.184717:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961404544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076995.184727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076995.184730:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076995.184734:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961404544:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076995.695862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076995.887908:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076996.719919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076996.912001:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076997.743845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769076997.935995:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076998.767849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076998.959955:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769076999.791837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769076999.983983:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077000.303854:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077000.303863:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077000.303867:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077000.303875:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2682 00000100:00080000:0.0:1769077000.303900:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077000.303906:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077000.303935:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961404672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077000.303962:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077000.303964:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077000.303966:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077000.303969:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2682 00000100:00080000:0.0:1769077000.303971:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077000.303975:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077000.303983:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077000.303993:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961404800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077000.304002:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961404672:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077000.304005:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077000.304011:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077000.304013:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077000.304017:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2682 00000100:00080000:0.0:1769077000.304019:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077000.304030:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077000.304034:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077000.304045:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961404928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077000.304122:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961404800:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077000.304164:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961404928:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077000.815909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077001.008071:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077001.839869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077002.031967:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077002.863869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077003.056049:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769077003.376041:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769077003.376088:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077003.376109:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077003.376196:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077003.376211:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769077003.376314:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077000/real 1769077003] req@ffff895e4383ed80 x1855008961404800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077003.376346:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961404800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077003.376362:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077003.376369:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077003.376375:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961404800:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077003.376387:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077000/real 1769077003] req@ffff895e4383d180 x1855008961404928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077003.376403:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961404928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077003.376441:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077003.376447:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077003.376468:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961404928:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077003.376489:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077000/real 1769077003] req@ffff895e4383df80 x1855008961404672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077003.376505:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961404672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077003.376515:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077003.376519:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077003.376523:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961404672:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077003.887858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077004.079999:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077004.911946:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077005.104019:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077005.423858:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077005.423870:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077005.423875:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077005.423887:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2687 00000100:00080000:2.0:1769077005.423899:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077005.423908:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077005.423945:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961405056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077005.424029:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077005.424034:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077005.424036:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077005.424042:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2687 00000100:00080000:2.0:1769077005.424048:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077005.424053:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077005.424064:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077005.424082:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961405184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077005.424094:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077005.424098:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077005.424100:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077005.424104:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2687 00000100:00080000:2.0:1769077005.424109:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077005.424114:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077005.424120:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077005.424132:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961405312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077005.424246:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961405056:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769077005.424327:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077005.424370:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077005/real 1769077005] req@ffff895e44e29f80 x1855008961405056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077005.424429:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961405056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077005.424441:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077005.424446:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077005.424451:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961405056:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077005.424460:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961405184:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077005.424490:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077005.424507:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077005/real 1769077005] req@ffff895e44e2b480 x1855008961405184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077005.424519:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961405184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077005.424530:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077005.424539:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077005.424543:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961405184:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077005.424550:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961405312:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077005.424573:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077005.424586:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077005/real 1769077005] req@ffff895e44e28e00 x1855008961405312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077005.424610:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961405312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077005.424620:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077005.424623:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077005.424627:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961405312:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077005.935863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077006.128013:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077006.959864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077007.151923:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077007.983868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077008.175951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077009.007857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077009.199898:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077010.031828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077010.223953:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077010.543819:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077010.543827:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077010.543831:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077010.543839:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2693 00000100:00080000:2.0:1769077010.543864:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077010.543870:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077010.543909:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961405440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077010.543931:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077010.543933:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077010.543935:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077010.543938:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2693 00000100:00080000:2.0:1769077010.543940:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077010.543942:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077010.543949:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077010.543956:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961405568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077010.543962:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077010.543964:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077010.543966:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077010.543968:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2693 00000100:00080000:2.0:1769077010.543969:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077010.543971:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077010.543974:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077010.543979:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961405696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077010.544053:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961405440:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769077010.544116:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077010.544150:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077010/real 1769077010] req@ffff895e44e29180 x1855008961405440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077010.544165:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961405440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077010.544182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077010.544185:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077010.544189:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961405440:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077010.544195:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961405568:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077010.544211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077010.544236:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077010/real 1769077010] req@ffff895e44e29500 x1855008961405568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077010.544252:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961405568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077010.544258:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077010.544260:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077010.544262:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961405568:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077010.544266:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961405696:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077010.544279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077010.544285:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077010/real 1769077010] req@ffff895e44e28a80 x1855008961405696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077010.544311:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961405696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077010.544317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077010.544319:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077010.544321:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961405696:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077011.055899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077011.247905:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077012.079835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077012.271920:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077013.103895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077013.295897:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077014.127915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077014.320044:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077015.151861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077015.343923:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077015.663875:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077015.663919:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077015.663925:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077015.663938:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2698 00000100:00080000:2.0:1769077015.663955:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077015.663964:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077015.664049:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961405824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077015.664103:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077015.664108:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077015.664111:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077015.664117:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2698 00000100:00080000:2.0:1769077015.664122:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077015.664129:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077015.664143:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077015.664162:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961405952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077015.664177:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077015.664182:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077015.664184:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077015.664189:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2698 00000100:00080000:2.0:1769077015.664195:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077015.664220:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961405824:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077015.664233:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077015.664240:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077015.664254:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961406080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077015.664326:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077015.664384:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077015/real 1769077015] req@ffff895e44e2bb80 x1855008961405824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077015.664416:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961405824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077015.664428:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077015.664433:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077015.664437:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961405824:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077015.664471:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961405952:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077015.664495:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077015.664507:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077015/real 1769077015] req@ffff895e44e2b800 x1855008961405952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077015.664519:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961405952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077015.664539:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077015.664542:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077015.664545:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961405952:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077015.664551:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961406080:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077015.664568:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077015.664577:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077015/real 1769077015] req@ffff895e44e29c00 x1855008961406080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077015.664594:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961406080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077015.664629:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077015.664633:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077015.664636:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961406080:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077016.175928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077016.367999:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077017.199912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077017.391948:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077018.223968:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077018.416052:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077019.247920:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077019.440071:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077020.271908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077020.464059:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077020.783951:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077020.783966:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077020.783972:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077020.783985:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2703 00000100:00080000:1.0:1769077020.784002:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077020.784012:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077020.784080:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961406208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077020.784136:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077020.784177:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077020.784180:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077020.784186:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2703 00000100:00080000:1.0:1769077020.784192:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077020.784197:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077020.784210:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077020.784220:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961406208:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077020.784234:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961406336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077020.784250:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077020.784254:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077020.784256:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077020.784261:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2703 00000100:00080000:1.0:1769077020.784264:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077020.784268:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077020.784274:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077020.784287:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961406464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077020.784325:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077020.784379:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077020/real 1769077020] req@ffff895e4713ce00 x1855008961406208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077020.784401:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961406208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077020.784414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077020.784419:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077020.784423:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961406208:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077020.784456:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961406336:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077020.784478:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077020.784490:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077020/real 1769077020] req@ffff895e4713f480 x1855008961406336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077020.784503:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961406336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077020.784512:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077020.784525:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077020.784529:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961406336:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077020.784537:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961406464:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077020.784558:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077020.784568:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077020/real 1769077020] req@ffff895e4713ca80 x1855008961406464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077020.784580:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961406464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077020.784588:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077020.784591:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077020.784594:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961406464:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077021.295867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077021.488009:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077022.319886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077022.512041:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077023.343889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077023.535927:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077024.367870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077024.560048:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077025.391927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077025.583952:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077025.903927:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077025.903939:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077025.903945:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077025.903955:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2708 00000100:00080000:0.0:1769077025.903967:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077025.903974:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077025.904021:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961406592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077025.904065:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077025.904068:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077025.904070:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077025.904073:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2708 00000100:00080000:0.0:1769077025.904076:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077025.904079:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077025.904088:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077025.904100:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961406720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077025.904107:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077025.904110:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077025.904112:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077025.904115:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2708 00000100:00080000:0.0:1769077025.904116:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077025.904120:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077025.904128:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961406592:192.168.204.117@tcp:250:kworker.0 00000008:00100000:0.0:1769077025.904146:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077025.904158:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961406848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077025.904199:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077025.904235:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077025/real 1769077025] req@ffff895e4383df80 x1855008961406592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077025.904250:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961406592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077025.904258:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077025.904262:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077025.904265:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961406592:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077025.904270:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961406720:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077025.904300:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077025.904309:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077025/real 1769077025] req@ffff895e4383d180 x1855008961406720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077025.904317:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961406720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077025.904332:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077025.904335:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077025.904338:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961406720:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077025.904359:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961406848:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077025.904372:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077025.904380:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077025/real 1769077025] req@ffff895e4383ed80 x1855008961406848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077025.904387:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961406848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077025.904393:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077025.904395:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077025.904397:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961406848:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077026.415878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077026.607980:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077027.439922:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077027.631891:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077028.463974:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077028.656001:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077029.487841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077029.679928:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077030.511872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077030.703992:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077031.024053:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077031.024071:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077031.024079:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077031.024096:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2713 00000100:00080000:0.0:1769077031.024116:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077031.024129:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077031.024189:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961406976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077031.024253:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077031.024258:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077031.024261:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077031.024274:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2713 00000100:00080000:0.0:1769077031.024280:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077031.024286:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077031.024301:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077031.024319:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961407104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077031.024351:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077031.024356:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961406976:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077031.024357:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077031.024386:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077031.024395:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2713 00000100:00080000:0.0:1769077031.024402:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077031.024409:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077031.024417:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077031.024435:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961407232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077031.024453:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077031.024531:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077031/real 1769077031] req@ffff895e4383d880 x1855008961406976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077031.024567:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961406976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077031.024577:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077031.024582:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077031.024586:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961406976:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077031.024614:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961407104:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077031.024638:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077031.024649:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077031/real 1769077031] req@ffff895e4383e680 x1855008961407104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077031.024660:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961407104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077031.024668:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077031.024670:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077031.024673:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961407104:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077031.024678:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961407232:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077031.024691:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077031.024698:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077031/real 1769077031] req@ffff895e4383c700 x1855008961407232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077031.024708:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961407232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077031.024736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077031.024739:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077031.024742:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961407232:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077031.535968:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077031.728043:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077032.559879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077032.751987:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077033.583872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077033.776042:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077034.607873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077034.799921:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077035.631887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077035.824064:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077036.143880:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077036.143910:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077036.143915:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077036.143927:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2718 00000100:00080000:2.0:1769077036.143942:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077036.143951:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077036.144005:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961407360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077036.144037:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077036.144041:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077036.144043:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077036.144046:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2718 00000100:00080000:2.0:1769077036.144050:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077036.144053:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077036.144063:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077036.144078:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961407488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077036.144087:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077036.144124:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077036.144126:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077036.144126:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961407360:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077036.144133:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2718 00000100:00080000:2.0:1769077036.144138:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077036.144141:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077036.144145:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077036.144154:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961407616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077036.144216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077036.144258:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077036/real 1769077036] req@ffff895e44e29c00 x1855008961407360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077036.144274:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961407360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077036.144282:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077036.144286:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077036.144289:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961407360:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077036.144294:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961407488:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077036.144309:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077036.144317:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077036/real 1769077036] req@ffff895e44e2b800 x1855008961407488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077036.144338:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961407488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077036.144344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077036.144347:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077036.144349:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961407488:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077036.144383:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961407616:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077036.144398:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077036.144405:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077036/real 1769077036] req@ffff895e44e2bb80 x1855008961407616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077036.144414:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961407616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077036.144419:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077036.144421:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077036.144424:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961407616:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077036.655919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077036.847971:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077037.679813:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077037.872063:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077038.703970:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077038.895976:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077039.728052:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077039.920008:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077040.751930:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077040.944004:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077041.263940:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077041.263950:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077041.263955:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077041.263965:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2723 00000100:00080000:3.0:1769077041.263976:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077041.263983:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077041.264020:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961407744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077041.264051:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077041.264055:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077041.264057:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077041.264060:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2723 00000100:00080000:3.0:1769077041.264063:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077041.264084:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077041.264093:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077041.264126:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961407872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077041.264135:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077041.264138:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077041.264139:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077041.264142:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2723 00000100:00080000:3.0:1769077041.264145:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077041.264150:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077041.264153:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961407744:192.168.204.117@tcp:250:kworker.0 00000008:00100000:3.0:1769077041.264155:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077041.264167:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961408000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077041.264246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077041.264311:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077041/real 1769077041] req@ffff895e453d7800 x1855008961407744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077041.264337:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961407744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077041.264377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077041.264382:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077041.264387:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961407744:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077041.264396:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961407872:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077041.264424:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077041.264437:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077041/real 1769077041] req@ffff895e453d5c00 x1855008961407872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077041.264470:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961407872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077041.264480:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077041.264484:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077041.264488:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961407872:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077041.264531:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961408000:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077041.264554:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077041.264565:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077041/real 1769077041] req@ffff895e453d4a80 x1855008961408000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077041.264576:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961408000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077041.264583:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077041.264586:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077041.264589:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961408000:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077041.775953:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077041.967917:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077042.799886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077042.991913:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077043.823882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077044.015889:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077044.847862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077045.039894:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077045.871909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077046.063941:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077046.383861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077046.383870:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077046.383874:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077046.383881:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2728 00000100:00080000:3.0:1769077046.383891:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077046.383896:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077046.383928:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961408128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077046.383951:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077046.383953:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077046.383968:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077046.383971:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2728 00000100:00080000:3.0:1769077046.383974:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077046.383977:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077046.383985:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077046.383999:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961408256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077046.384001:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961408128:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077046.384015:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077046.384018:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077046.384019:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077046.384022:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2728 00000100:00080000:3.0:1769077046.384025:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077046.384027:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077046.384031:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077046.384039:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961408384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077046.384066:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077046.384098:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077046/real 1769077046] req@ffff895e453d5f80 x1855008961408128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077046.384112:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961408128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077046.384126:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077046.384129:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077046.384132:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961408128:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077046.384151:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961408256:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077046.384170:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077046.384182:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077046/real 1769077046] req@ffff895e453d4a80 x1855008961408256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077046.384195:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961408256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077046.384204:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077046.384207:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077046.384210:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961408256:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077046.384217:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961408384:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077046.384237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077046.384247:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077046/real 1769077046] req@ffff895e453d5c00 x1855008961408384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077046.384254:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961408384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077046.384260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077046.384262:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077046.384264:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961408384:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077046.895909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077047.087997:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077047.919905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077048.111926:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077048.943852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077049.135859:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077049.967965:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077050.159997:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077050.991899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077051.184073:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077051.503907:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077051.503919:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077051.503923:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077051.503934:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2734 00000100:00080000:2.0:1769077051.503947:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077051.503954:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077051.503998:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961408512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077051.504027:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077051.504030:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077051.504031:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077051.504034:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2734 00000100:00080000:2.0:1769077051.504037:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077051.504039:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077051.504049:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077051.504082:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961408640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077051.504084:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961408512:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077051.504109:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077051.504112:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077051.504115:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077051.504120:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2734 00000100:00080000:2.0:1769077051.504124:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077051.504128:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077051.504132:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077051.504140:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961408768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077051.504176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077051.504225:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077051/real 1769077051] req@ffff895e44e28a80 x1855008961408512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077051.504245:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961408512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077051.504255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077051.504260:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077051.504264:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961408512:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077051.504291:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961408640:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077051.504315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077051.504328:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077051/real 1769077051] req@ffff895e44e29500 x1855008961408640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077051.504340:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961408640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077051.504348:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077051.504352:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077051.504356:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961408640:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077051.504363:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961408768:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077051.504381:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077051.504406:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077051/real 1769077051] req@ffff895e44e29180 x1855008961408768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077051.504414:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961408768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077051.504419:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077051.504422:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077051.504424:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961408768:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077052.015883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077052.208038:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077053.039818:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077053.231987:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077054.063836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077054.255949:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077055.087824:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077055.280012:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077056.111926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077056.303968:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077056.623920:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077056.623944:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077056.623949:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077056.623961:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2739 00000100:00080000:2.0:1769077056.623973:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077056.623980:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077056.624015:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961408896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077056.624054:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077056.624057:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077056.624059:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077056.624062:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2739 00000100:00080000:2.0:1769077056.624081:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077056.624085:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077056.624094:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077056.624102:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961408896:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077056.624109:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961409024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077056.624131:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077056.624134:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077056.624136:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077056.624139:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2739 00000100:00080000:2.0:1769077056.624142:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077056.624145:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077056.624150:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077056.624158:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961409152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077056.624177:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077056.624229:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077056/real 1769077056] req@ffff895e44e28e00 x1855008961408896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077056.624262:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961408896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077056.624275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077056.624280:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077056.624285:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961408896:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077056.624314:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961409024:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077056.624340:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077056.624352:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077056/real 1769077056] req@ffff895e44e2b480 x1855008961409024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077056.624362:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961409024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077056.624388:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077056.624391:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077056.624394:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961409024:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077056.624399:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961409152:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077056.624415:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077056.624422:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077056/real 1769077056] req@ffff895e44e29f80 x1855008961409152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077056.624431:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961409152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077056.624438:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077056.624440:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077056.624442:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961409152:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077057.135882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077057.327982:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077058.159858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077058.351935:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077059.183883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077059.375891:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077060.207830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077060.399963:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077061.231925:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077061.424051:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077061.743873:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077061.743882:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077061.743887:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077061.743900:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2744 00000100:00080000:0.0:1769077061.743912:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077061.743919:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077061.743952:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961409280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077061.743980:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077061.743998:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077061.744000:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077061.744004:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2744 00000100:00080000:0.0:1769077061.744008:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077061.744030:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077061.744042:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077061.744059:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961409408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077061.744074:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961409280:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077061.744084:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077061.744090:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077061.744092:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077061.744100:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2744 00000100:00080000:0.0:1769077061.744107:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077061.744113:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077061.744120:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077061.744134:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961409536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077061.744166:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077061.744215:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077061/real 1769077061] req@ffff895e4383fb80 x1855008961409280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077061.744239:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961409280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077061.744252:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077061.744275:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077061.744279:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961409280:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077061.744307:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961409408:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077061.744328:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077061.744338:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077061/real 1769077061] req@ffff895e4383f100 x1855008961409408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077061.744347:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961409408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077061.744353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077061.744356:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077061.744358:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961409408:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077061.744363:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961409536:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077061.744376:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077061.744382:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077061/real 1769077061] req@ffff895e4383c380 x1855008961409536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077061.744390:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961409536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077061.744395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077061.744397:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077061.744399:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961409536:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077062.255837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077062.447934:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077063.279838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077063.472002:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077064.303915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077064.496023:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077065.327916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077065.520022:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077066.351852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077066.543894:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077066.863864:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077066.863875:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077066.863879:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077066.863887:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2749 00000100:00080000:2.0:1769077066.863910:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077066.863916:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077066.863944:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961409664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077066.863967:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077066.863970:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077066.863971:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077066.863974:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2749 00000100:00080000:2.0:1769077066.863976:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077066.863978:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077066.863986:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077066.863994:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961409792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077066.864003:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077066.864005:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077066.864007:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077066.864011:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961409664:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077066.864012:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2749 00000100:00080000:2.0:1769077066.864014:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077066.864017:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077066.864020:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077066.864032:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961409920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077066.864132:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961409792:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077066.864171:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961409920:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077067.375824:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077067.567982:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077068.399826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077068.591924:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077069.423878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077069.615899:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769077069.936044:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769077069.936082:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077069.936098:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769077069.936135:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769077069.936144:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769077069.936263:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077066/real 1769077069] req@ffff895e44e2a300 x1855008961409792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077069.936289:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961409792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077069.936329:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077069.936335:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077069.936340:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961409792:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077069.936348:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077066/real 1769077069] req@ffff895e44e2aa00 x1855008961409920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077069.936363:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961409920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077069.936370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077069.936374:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077069.936378:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961409920:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077069.936385:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077066/real 1769077069] req@ffff895e44e2b100 x1855008961409664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077069.936399:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961409664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077069.936433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077069.936437:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077069.936441:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961409664:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077070.447821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077070.639958:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077071.471839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077071.663882:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077071.983942:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077071.983978:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077071.983984:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077071.983997:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2754 00000100:00080000:1.0:1769077071.984009:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077071.984017:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077071.984065:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961410048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077071.984092:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077071.984095:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077071.984096:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077071.984099:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2754 00000100:00080000:1.0:1769077071.984102:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077071.984105:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077071.984114:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077071.984123:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961410176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077071.984130:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077071.984133:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077071.984134:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077071.984137:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2754 00000100:00080000:1.0:1769077071.984138:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077071.984141:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077071.984143:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077071.984149:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961410304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077071.984196:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961410048:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769077071.984342:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077071.984419:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077071/real 1769077071] req@ffff895e4713f800 x1855008961410048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077071.984446:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961410048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077071.984461:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077071.984468:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077071.984473:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961410048:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077071.984483:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961410176:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077071.984518:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077071.984543:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077071/real 1769077071] req@ffff895e4713dc00 x1855008961410176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077071.984559:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961410176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077071.984569:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077071.984573:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077071.984577:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961410176:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077071.984594:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961410304:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077071.984619:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077071.984632:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077071/real 1769077071] req@ffff895e4713d180 x1855008961410304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077071.984779:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961410304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077071.984790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077071.984794:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077071.984798:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961410304:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077072.495855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077072.687969:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077073.519846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077073.711993:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077074.543829:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077074.735948:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077075.567843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077075.759896:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077076.591852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077076.783910:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077077.103906:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077077.103918:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077077.103923:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077077.103934:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2759 00000100:00080000:3.0:1769077077.103945:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077077.103951:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077077.103982:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961410432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077077.104008:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077077.104011:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077077.104012:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077077.104015:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2759 00000100:00080000:3.0:1769077077.104018:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077077.104036:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077077.104045:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077077.104051:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961410432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077077.104057:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961410560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077077.104066:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077077.104069:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077077.104071:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077077.104074:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2759 00000100:00080000:3.0:1769077077.104077:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077077.104079:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077077.104082:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077077.104090:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961410688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077077.104113:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077077.104168:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077077/real 1769077077] req@ffff895e453d7800 x1855008961410432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077077.104201:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961410432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077077.104220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077077.104223:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077077.104226:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961410432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077077.104251:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961410560:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077077.104271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077077.104280:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077077/real 1769077077] req@ffff895e453d4e00 x1855008961410560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077077.104289:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961410560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077077.104295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077077.104297:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077077.104299:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961410560:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077077.104303:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961410688:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077077.104313:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077077.104319:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077077/real 1769077077] req@ffff895e453d7480 x1855008961410688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077077.104326:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961410688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077077.104331:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077077.104333:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077077.104334:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961410688:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077077.615866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077077.807898:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077078.639862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077078.831962:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077079.663836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077079.855861:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077080.687982:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077080.879907:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077081.711856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077081.903864:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077082.223856:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077082.223866:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077082.223870:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077082.223878:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2764 00000100:00080000:3.0:1769077082.223887:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077082.223893:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077082.223925:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961410816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077082.223948:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077082.223951:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077082.223966:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077082.223969:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2764 00000100:00080000:3.0:1769077082.223973:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077082.223976:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077082.223984:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077082.223993:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961410944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077082.224002:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077082.224004:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077082.224008:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077082.224010:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961410816:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077082.224012:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2764 00000100:00080000:3.0:1769077082.224017:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077082.224022:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077082.224025:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077082.224032:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961411072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077082.224111:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077082.224152:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077082/real 1769077082] req@ffff895e453d5500 x1855008961410816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077082.224173:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961410816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077082.224185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077082.224189:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077082.224194:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961410816:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077082.224217:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961410944:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077082.224240:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077082.224253:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077082/real 1769077082] req@ffff895e453d7b80 x1855008961410944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077082.224267:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961410944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077082.224276:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077082.224279:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077082.224283:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961410944:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077082.224290:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961411072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077082.224311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077082.224323:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077082/real 1769077082] req@ffff895e453d7100 x1855008961411072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077082.224345:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961411072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077082.224354:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077082.224358:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077082.224361:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961411072:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077082.735843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077082.927981:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077083.759839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077083.951875:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077084.783866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077084.975970:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077085.807847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077085.999883:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077086.831844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077087.023925:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077087.343886:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077087.343894:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077087.343898:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077087.343906:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2769 00000100:00080000:0.0:1769077087.343916:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077087.343922:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077087.343954:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961411200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077087.343977:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077087.343980:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077087.343981:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077087.343984:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2769 00000100:00080000:0.0:1769077087.343986:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077087.343989:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077087.344017:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077087.344029:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961411328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077087.344036:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961411200:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077087.344059:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077087.344062:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077087.344064:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077087.344067:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2769 00000100:00080000:0.0:1769077087.344071:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077087.344074:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077087.344079:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077087.344087:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961411456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077087.344107:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077087.344145:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077087/real 1769077087] req@ffff895e4383f480 x1855008961411200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077087.344160:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961411200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077087.344168:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077087.344179:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077087.344182:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961411200:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077087.344200:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961411328:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077087.344217:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077087.344226:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077087/real 1769077087] req@ffff895e4383c000 x1855008961411328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077087.344234:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961411328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077087.344239:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077087.344242:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077087.344244:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961411328:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077087.344247:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961411456:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077087.344257:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077087.344262:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077087/real 1769077087] req@ffff895e4383ce00 x1855008961411456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077087.344270:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961411456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077087.344274:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077087.344276:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077087.344278:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961411456:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077087.855834:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077088.047877:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077088.879860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077089.071905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077089.903861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077090.096016:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077090.927909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077091.120018:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077091.951832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077092.143913:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077092.463865:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077092.463877:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077092.463882:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077092.463893:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2775 00000100:00080000:2.0:1769077092.463905:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077092.463936:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077092.463980:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961411584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077092.464034:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077092.464038:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077092.464040:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077092.464045:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2775 00000100:00080000:2.0:1769077092.464050:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077092.464057:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077092.464069:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077092.464082:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961411712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077092.464093:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077092.464097:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769077092.464099:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961411584:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077092.464100:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077092.464104:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2775 00000100:00080000:2.0:1769077092.464107:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077092.464110:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077092.464114:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077092.464125:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961411840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077092.464196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077092.464233:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077092/real 1769077092] req@ffff895e44e28000 x1855008961411584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077092.464247:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961411584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077092.464256:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077092.464259:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077092.464263:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961411584:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077092.464290:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961411712:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077092.464305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077092.464314:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077092/real 1769077092] req@ffff895e44e28380 x1855008961411712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077092.464323:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961411712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077092.464329:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077092.464331:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077092.464333:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961411712:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077092.464338:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961411840:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077092.464348:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077092.464354:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077092/real 1769077092] req@ffff895e44e29880 x1855008961411840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077092.464372:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961411840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077092.464378:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077092.464380:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077092.464382:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961411840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077092.975901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077093.167966:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077093.999849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077094.191975:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077095.023863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077095.216021:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077096.047868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077096.239932:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077097.071886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077097.263970:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077097.583920:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077097.583958:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077097.583962:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077097.583973:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2780 00000100:00080000:2.0:1769077097.583986:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077097.583993:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077097.584036:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961411968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077097.584068:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077097.584071:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077097.584073:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077097.584077:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2780 00000100:00080000:2.0:1769077097.584081:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077097.584085:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077097.584096:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077097.584112:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961412096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077097.584125:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961411968:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077097.584126:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077097.584137:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077097.584139:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077097.584143:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2780 00000100:00080000:2.0:1769077097.584146:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077097.584150:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077097.584155:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077097.584165:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961412224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077097.584207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077097.584249:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077097/real 1769077097] req@ffff895e44e29880 x1855008961411968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077097.584264:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961411968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077097.584273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077097.584276:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077097.584279:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961411968:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077097.584302:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961412096:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077097.584317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077097.584325:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077097/real 1769077097] req@ffff895e44e28380 x1855008961412096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077097.584341:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961412096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077097.584348:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077097.584350:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077097.584352:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961412096:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077097.584356:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961412224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077097.584369:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077097.584375:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077097/real 1769077097] req@ffff895e44e28000 x1855008961412224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077097.584383:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961412224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077097.584388:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077097.584390:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077097.584392:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961412224:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077098.095860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077098.287975:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077099.119877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077099.311941:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077100.143855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077100.335993:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077101.167868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077101.359961:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077102.191894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077102.383996:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077102.703823:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077102.703835:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077102.703839:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077102.703849:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2785 00000100:00080000:2.0:1769077102.703863:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077102.703870:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077102.703915:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961412352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077102.703946:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077102.703949:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077102.703950:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077102.703953:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2785 00000100:00080000:2.0:1769077102.703956:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077102.703958:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077102.703967:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077102.703978:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961412480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077102.703986:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077102.703989:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077102.703991:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077102.704008:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961412352:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077102.704019:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2785 00000100:00080000:2.0:1769077102.704024:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077102.704029:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077102.704035:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077102.704052:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961412608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077102.704087:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077102.704126:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077102/real 1769077102] req@ffff895e44e2b100 x1855008961412352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077102.704154:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961412352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077102.704162:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077102.704166:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077102.704169:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961412352:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077102.704175:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961412480:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077102.704192:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077102.704200:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077102/real 1769077102] req@ffff895e44e2aa00 x1855008961412480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077102.704208:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961412480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077102.704214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077102.704217:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077102.704219:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961412480:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077102.704242:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961412608:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077102.704253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077102.704263:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077102/real 1769077102] req@ffff895e44e2a300 x1855008961412608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077102.704275:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961412608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077102.704297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077102.704301:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077102.704305:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961412608:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077103.215851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077103.407917:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077104.239934:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077104.431895:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077105.263888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077105.455895:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077106.287861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077106.479920:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077107.311835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077107.503922:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077107.823898:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077107.823912:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077107.823917:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077107.823928:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2790 00000100:00080000:3.0:1769077107.823956:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077107.823963:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077107.824008:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961412736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077107.824039:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077107.824042:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077107.824044:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077107.824046:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2790 00000100:00080000:3.0:1769077107.824049:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077107.824051:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077107.824060:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077107.824068:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961412864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077107.824076:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077107.824100:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077107.824102:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077107.824106:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2790 00000100:00080000:3.0:1769077107.824109:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077107.824114:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961412736:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077107.824115:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077107.824122:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077107.824132:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961412992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077107.824209:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077107.824261:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077107/real 1769077107] req@ffff895e453d4700 x1855008961412736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077107.824285:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961412736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077107.824296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077107.824301:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077107.824305:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961412736:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077107.824314:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961412864:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077107.824338:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077107.824352:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077107/real 1769077107] req@ffff895e453d5880 x1855008961412864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077107.824373:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961412864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077107.824399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077107.824403:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077107.824407:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961412864:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077107.824436:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961412992:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077107.824477:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077107.824488:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077107/real 1769077107] req@ffff895e453d5180 x1855008961412992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077107.824501:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961412992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077107.824509:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077107.824513:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077107.824517:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961412992:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077108.335856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077108.527936:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077109.359809:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077109.551944:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077110.383865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077110.575968:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077111.407955:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077111.599899:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077112.431830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077112.623887:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077112.943892:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077112.943903:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077112.943908:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077112.943918:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2795 00000100:00080000:0.0:1769077112.943930:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077112.943936:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077112.943988:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961413120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077112.944020:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077112.944024:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077112.944026:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077112.944031:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2795 00000100:00080000:0.0:1769077112.944036:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077112.944042:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077112.944053:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077112.944053:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961413120:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077112.944072:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961413248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077112.944084:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077112.944087:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077112.944089:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077112.944093:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2795 00000100:00080000:0.0:1769077112.944098:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077112.944102:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077112.944107:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077112.944115:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077112.944122:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961413376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077112.944152:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077112/real 1769077112] req@ffff895e4383dc00 x1855008961413120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077112.944169:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961413120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077112.944177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077112.944181:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077112.944184:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961413120:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077112.944203:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961413248:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077112.944224:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077112.944237:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077112/real 1769077112] req@ffff895e4383ca80 x1855008961413248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077112.944250:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961413248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077112.944266:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077112.944269:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077112.944271:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961413248:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077112.944276:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961413376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077112.944293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077112.944301:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077112/real 1769077112] req@ffff895e4383ea00 x1855008961413376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077112.944309:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961413376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077112.944314:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077112.944316:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077112.944318:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961413376:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077113.455861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077113.647955:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077114.479852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077114.671886:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077115.503853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077115.695912:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077116.527945:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077116.719903:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077117.551879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077117.743916:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077118.063857:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077118.063871:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077118.063877:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077118.063890:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2800 00000100:00080000:0.0:1769077118.063905:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077118.063913:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077118.063966:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961413504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077118.064000:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077118.064003:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077118.064005:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077118.064009:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2800 00000100:00080000:0.0:1769077118.064012:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077118.064016:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077118.064027:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077118.064038:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961413632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077118.064048:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077118.064051:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077118.064053:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077118.064057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2800 00000100:00080000:0.0:1769077118.064059:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077118.064063:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077118.064091:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077118.064107:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961413760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077118.064107:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961413504:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769077118.064207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077118.064272:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077118/real 1769077118] req@ffff895e4383f800 x1855008961413504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077118.064292:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961413504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077118.064305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077118.064314:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077118.064333:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961413504:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077118.064344:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961413632:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077118.064379:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077118.064397:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077118/real 1769077118] req@ffff895e4383e300 x1855008961413632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077118.064416:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961413632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077118.064425:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077118.064429:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077118.064432:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961413632:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077118.064467:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961413760:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077118.064537:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077118.064554:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077118/real 1769077118] req@ffff895e4383d500 x1855008961413760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077118.064567:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961413760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077118.064575:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077118.064578:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077118.064581:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961413760:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077118.575838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077118.767903:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077119.599851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077119.791899:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077120.623837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077120.815864:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077121.647850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077121.839928:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077122.671832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077122.863856:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077123.183862:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077123.183871:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077123.183875:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077123.183898:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2805 00000100:00080000:1.0:1769077123.183910:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077123.183916:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077123.183955:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961413888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077123.183995:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077123.183998:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077123.184000:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077123.184004:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2805 00000100:00080000:1.0:1769077123.184007:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077123.184011:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077123.184020:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077123.184030:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961414016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077123.184040:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077123.184043:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077123.184045:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077123.184063:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2805 00000100:00080000:1.0:1769077123.184079:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077123.184082:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077123.184086:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077123.184095:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961414144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077123.184119:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961413888:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769077123.184180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077123.184219:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077123/real 1769077123] req@ffff895e4713df80 x1855008961413888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077123.184233:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961413888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077123.184241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077123.184245:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077123.184248:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961413888:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077123.184254:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961414016:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077123.184270:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077123.184277:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077123/real 1769077123] req@ffff895e4713c380 x1855008961414016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077123.184285:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961414016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077123.184291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077123.184293:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077123.184295:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961414016:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077123.184298:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961414144:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077123.184308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077123.184313:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077123/real 1769077123] req@ffff895e4713d180 x1855008961414144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077123.184320:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961414144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077123.184334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077123.184336:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077123.184339:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961414144:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077123.695861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077123.887919:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077124.719876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077124.911951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077125.743892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077125.936018:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077126.767891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077126.959969:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077127.791886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077127.983934:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077128.303928:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077128.303940:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077128.303944:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077128.303956:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2810 00000100:00080000:2.0:1769077128.303969:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077128.303976:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077128.304022:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961414272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077128.304049:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077128.304052:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077128.304053:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077128.304057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2810 00000100:00080000:2.0:1769077128.304059:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077128.304061:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077128.304071:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077128.304080:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961414400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077128.304088:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077128.304091:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077128.304092:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077128.304095:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2810 00000100:00080000:2.0:1769077128.304097:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077128.304099:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077128.304107:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961414272:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769077128.304124:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077128.304136:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961414528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077128.304195:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077128.304245:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077128/real 1769077128] req@ffff895e44e29f80 x1855008961414272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077128.304263:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961414272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077128.304274:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077128.304277:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077128.304280:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961414272:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077128.304286:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961414400:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077128.304304:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077128.304336:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077128/real 1769077128] req@ffff895e44e2b480 x1855008961414400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077128.304346:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961414400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077128.304352:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077128.304355:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077128.304357:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961414400:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077128.304393:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961414528:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077128.304410:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077128.304417:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077128/real 1769077128] req@ffff895e44e28e00 x1855008961414528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077128.304425:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961414528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077128.304431:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077128.304433:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077128.304435:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961414528:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077128.815847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077129.007991:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077129.839866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077130.031901:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077130.863808:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077131.055911:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077131.887853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077132.079915:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077132.911843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077133.103904:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077133.423821:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077133.423831:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077133.423836:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077133.423845:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2815 00000100:00080000:2.0:1769077133.423856:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077133.423863:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077133.423898:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961414656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077133.423927:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077133.423931:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077133.423939:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077133.423943:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2815 00000100:00080000:2.0:1769077133.423948:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077133.423952:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077133.423961:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077133.423978:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961414784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077133.423978:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961414656:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077133.423995:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077133.424014:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077133.424017:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077133.424021:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2815 00000100:00080000:2.0:1769077133.424039:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077133.424044:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077133.424049:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077133.424063:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961414912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077133.424092:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961414784:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077133.424118:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961414912:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077133.935858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077134.127884:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077134.959942:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077135.152049:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077135.983866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077136.175921:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769077136.496098:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769077136.496150:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077136.496170:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077136.496231:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077136.496243:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769077136.496294:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077133/real 1769077136] req@ffff895e44e29500 x1855008961414784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077136.496322:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961414784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077136.496337:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077136.496344:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077136.496349:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961414784:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077136.496359:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077133/real 1769077136] req@ffff895e44e28a80 x1855008961414912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077136.496397:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961414912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077136.496406:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077136.496410:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077136.496413:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961414912:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077136.496420:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077133/real 1769077136] req@ffff895e44e29180 x1855008961414656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077136.496441:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961414656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077136.496448:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077136.496453:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077136.496456:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961414656:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077137.007841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077137.199946:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077138.031846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077138.223938:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077138.543890:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077138.543903:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077138.543907:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077138.543918:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2821 00000100:00080000:3.0:1769077138.543930:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077138.543938:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077138.543975:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961415040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077138.544010:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077138.544029:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077138.544031:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077138.544036:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2821 00000100:00080000:3.0:1769077138.544041:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077138.544047:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077138.544056:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961415040:192.168.204.117@tcp:250:kworker.0 00000008:00100000:3.0:1769077138.544057:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077138.544077:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961415168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077138.544090:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077138.544093:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077138.544095:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077138.544100:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2821 00000100:00080000:3.0:1769077138.544105:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077138.544110:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077138.544115:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077138.544128:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961415296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077138.544156:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077138.544199:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077138/real 1769077138] req@ffff895e453d6680 x1855008961415040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077138.544219:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961415040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077138.544236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077138.544242:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077138.544247:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961415040:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077138.544281:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961415168:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077138.544309:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077138.544324:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077138/real 1769077138] req@ffff895e453d4380 x1855008961415168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077138.544339:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961415168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077138.544349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077138.544353:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077138.544357:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961415168:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077138.544364:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961415296:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077138.544390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077138.544403:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077138/real 1769077138] req@ffff895e453d6d80 x1855008961415296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077138.544416:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961415296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077138.544426:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077138.544430:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077138.544434:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961415296:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077139.055873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077139.247925:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077140.079848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077140.271903:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077141.103844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077141.295894:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077142.127845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077142.319882:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077143.151905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077143.343967:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077143.663885:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077143.663900:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077143.663906:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077143.663921:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2826 00000100:00080000:3.0:1769077143.663936:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077143.663945:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077143.663989:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961415424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077143.664076:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077143.664081:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077143.664083:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077143.664090:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2826 00000100:00080000:3.0:1769077143.664096:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077143.664102:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077143.664115:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077143.664121:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961415424:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077143.664145:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961415552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077143.664159:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077143.664162:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077143.664164:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077143.664167:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2826 00000100:00080000:3.0:1769077143.664171:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077143.664174:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077143.664177:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077143.664186:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961415680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077143.664201:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077143.664245:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077143/real 1769077143] req@ffff895e453d6300 x1855008961415424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077143.664261:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961415424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077143.664270:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077143.664273:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077143.664277:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961415424:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077143.664301:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961415552:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077143.664317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077143.664326:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077143/real 1769077143] req@ffff895e453d6d80 x1855008961415552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077143.664334:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961415552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077143.664340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077143.664343:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077143.664345:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961415552:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077143.664357:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961415680:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077143.664371:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077143.664378:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077143/real 1769077143] req@ffff895e453d4380 x1855008961415680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077143.664386:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961415680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077143.664392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077143.664394:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077143.664395:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961415680:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077144.175875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077144.367954:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077145.199910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077145.391890:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077146.223882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077146.415911:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077147.247899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077147.439890:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077148.271840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077148.464003:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077148.783872:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077148.783881:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077148.783885:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077148.783894:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2831 00000100:00080000:3.0:1769077148.783930:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077148.783936:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077148.783983:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961415808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077148.784041:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077148.784044:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077148.784046:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077148.784049:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2831 00000100:00080000:3.0:1769077148.784052:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077148.784056:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077148.784064:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077148.784075:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961415936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077148.784083:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077148.784086:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077148.784087:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077148.784091:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2831 00000100:00080000:3.0:1769077148.784093:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077148.784096:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077148.784099:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077148.784106:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961415808:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077148.784109:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961416064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077148.784186:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077148.784231:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077148/real 1769077148] req@ffff895e453d6680 x1855008961415808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077148.784245:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961415808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077148.784253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077148.784257:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077148.784260:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961415808:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077148.784265:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961415936:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077148.784280:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077148.784288:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077148/real 1769077148] req@ffff895e453d5180 x1855008961415936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077148.784316:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961415936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077148.784322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077148.784325:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077148.784327:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961415936:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077148.784346:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961416064:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077148.784360:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077148.784366:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077148/real 1769077148] req@ffff895e453d5880 x1855008961416064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077148.784374:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961416064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077148.784409:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077148.784411:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077148.784414:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961416064:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077149.295880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077149.487941:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077150.319829:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077150.511909:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077151.343850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077151.535910:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077152.367883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077152.560025:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077153.391833:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077153.583954:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077153.903841:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077153.903872:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077153.903877:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077153.903907:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2836 00000100:00080000:3.0:1769077153.903919:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077153.903928:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077153.903967:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961416192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077153.904013:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077153.904018:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077153.904021:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077153.904026:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2836 00000100:00080000:3.0:1769077153.904031:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077153.904037:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077153.904049:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077153.904067:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961416192:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077153.904073:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961416320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077153.904091:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077153.904096:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077153.904099:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077153.904105:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2836 00000100:00080000:3.0:1769077153.904123:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077153.904140:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:0.0:1769077153.904144:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:3.0:1769077153.904152:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077153.904173:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961416448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077153.904185:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077153/real 1769077153] req@ffff895e453d4700 x1855008961416192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077153.904226:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961416192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077153.904249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077153.904253:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077153.904257:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961416192:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077153.904284:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961416320:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077153.904315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077153.904333:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077153/real 1769077153] req@ffff895e453d7100 x1855008961416320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077153.904348:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961416320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077153.904358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077153.904362:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077153.904366:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961416320:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077153.904374:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961416448:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077153.904396:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077153.904408:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077153/real 1769077153] req@ffff895e453d7b80 x1855008961416448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077153.904431:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961416448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077153.904442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077153.904446:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077153.904451:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961416448:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077154.415838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077154.607918:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077155.439826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077155.631959:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077156.463935:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077156.656021:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077157.487932:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077157.679945:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077158.511923:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077158.704099:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077159.023886:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077159.023897:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077159.023902:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077159.023911:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2841 00000100:00080000:2.0:1769077159.023924:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077159.023931:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077159.023976:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961416576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077159.024002:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077159.024005:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077159.024006:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077159.024010:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2841 00000100:00080000:2.0:1769077159.024012:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077159.024015:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077159.024024:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077159.024035:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961416704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077159.024044:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961416576:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077159.024044:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077159.024051:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077159.024054:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077159.024057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2841 00000100:00080000:2.0:1769077159.024059:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077159.024078:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077159.024082:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077159.024093:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961416832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077159.024128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077159.024174:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077159/real 1769077159] req@ffff895e44e2bb80 x1855008961416576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077159.024191:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961416576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077159.024199:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077159.024202:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077159.024205:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961416576:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077159.024230:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961416704:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077159.024246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077159.024255:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077159/real 1769077159] req@ffff895e44e2b800 x1855008961416704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077159.024263:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961416704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077159.024268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077159.024270:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077159.024272:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961416704:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077159.024284:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961416832:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077159.024298:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077159.024304:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077159/real 1769077159] req@ffff895e44e29c00 x1855008961416832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077159.024312:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961416832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077159.024317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077159.024319:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077159.024320:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961416832:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077159.535876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077159.727991:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077160.559938:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077160.751968:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077161.583857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077161.775926:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077162.607879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077162.799963:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077163.631911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077163.823999:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077164.143962:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077164.144020:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077164.144027:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077164.144042:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2846 00000100:00080000:1.0:1769077164.144060:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077164.144070:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077164.144125:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961416960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077164.144181:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077164.144186:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077164.144189:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077164.144194:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2846 00000100:00080000:1.0:1769077164.144199:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077164.144203:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077164.144217:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077164.144233:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961417088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077164.144248:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077164.144253:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077164.144256:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077164.144266:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961416960:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077164.144267:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2846 00000100:00080000:1.0:1769077164.144276:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077164.144286:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077164.144291:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077164.144306:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961417216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077164.144409:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077164.144488:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077164/real 1769077164] req@ffff895e4713dc00 x1855008961416960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077164.144534:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961416960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077164.144549:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077164.144555:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077164.144572:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961416960:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077164.144623:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961417088:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077164.144659:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077164.144677:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077164/real 1769077164] req@ffff895e4713f800 x1855008961417088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077164.144693:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961417088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077164.144705:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077164.144819:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077164.144825:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961417088:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077164.144835:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961417216:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077164.144868:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077164.144884:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077164/real 1769077164] req@ffff895e4713ca80 x1855008961417216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077164.144900:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961417216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077164.144911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077164.144915:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077164.144919:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961417216:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077164.655875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077164.848025:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077165.679858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077165.871991:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077166.703928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077166.895893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077167.727836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077167.920020:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077168.751912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077168.943937:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077169.263919:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077169.263928:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077169.263931:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077169.263939:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2851 00000100:00080000:2.0:1769077169.263950:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077169.263956:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077169.263990:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961417344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077169.264018:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077169.264021:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077169.264022:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077169.264025:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2851 00000100:00080000:2.0:1769077169.264027:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077169.264030:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077169.264036:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077169.264061:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961417472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077169.264069:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077169.264073:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961417344:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077169.264073:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077169.264077:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077169.264085:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2851 00000100:00080000:2.0:1769077169.264088:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077169.264090:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077169.264094:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077169.264103:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961417600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077169.264138:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077169.264172:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077169/real 1769077169] req@ffff895e44e29c00 x1855008961417344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077169.264185:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961417344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077169.264193:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077169.264196:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077169.264199:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961417344:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077169.264219:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961417472:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077169.264233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077169.264241:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077169/real 1769077169] req@ffff895e44e2b800 x1855008961417472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077169.264249:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961417472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077169.264255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077169.264257:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077169.264259:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961417472:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077169.264263:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961417600:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077169.264272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077169.264278:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077169/real 1769077169] req@ffff895e44e2bb80 x1855008961417600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077169.264293:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961417600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077169.264298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077169.264301:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077169.264303:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961417600:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077169.775889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077169.967959:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077170.799906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077170.991894:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077171.823884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077172.015908:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077172.847845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077173.039860:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077173.871878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077174.063998:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077174.383866:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077174.383876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077174.383881:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077174.383891:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2856 00000100:00080000:1.0:1769077174.383902:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077174.383909:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077174.383965:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961417728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077174.383997:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077174.384001:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077174.384003:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077174.384007:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2856 00000100:00080000:1.0:1769077174.384011:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077174.384025:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077174.384034:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077174.384046:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961417856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077174.384057:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077174.384060:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077174.384062:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077174.384101:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2856 00000100:00080000:1.0:1769077174.384106:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077174.384110:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077174.384115:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077174.384125:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961417728:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077174.384129:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961417984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077174.384219:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077174.384285:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077174/real 1769077174] req@ffff895e4713f480 x1855008961417728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077174.384309:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961417728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077174.384323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077174.384328:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077174.384334:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961417728:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077174.384343:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961417856:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077174.384369:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077174.384383:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077174/real 1769077174] req@ffff895e4713ce00 x1855008961417856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077174.384413:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961417856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077174.384423:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077174.384428:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077174.384432:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961417856:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077174.384461:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961417984:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077174.384503:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077174.384517:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077174/real 1769077174] req@ffff895e4713d500 x1855008961417984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077174.384555:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961417984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077174.384586:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077174.384592:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077174.384597:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961417984:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077174.895862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077175.087965:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077175.919893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077176.111940:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077176.943899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077177.135949:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077177.967862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077178.159956:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077178.991865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077179.183867:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077179.503904:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077179.503913:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077179.503918:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077179.503926:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2862 00000100:00080000:3.0:1769077179.503936:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077179.503942:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077179.503977:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961418112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077179.504002:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077179.504005:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077179.504006:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077179.504010:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2862 00000100:00080000:3.0:1769077179.504013:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077179.504015:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077179.504023:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077179.504040:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961418240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077179.504066:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077179.504069:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077179.504071:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077179.504075:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961418112:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077179.504075:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2862 00000100:00080000:3.0:1769077179.504079:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077179.504085:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077179.504102:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077179.504112:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961418368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077179.504149:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077179.504198:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077179/real 1769077179] req@ffff895e453d5500 x1855008961418112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077179.504219:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961418112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077179.504230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077179.504234:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077179.504239:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961418112:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077179.504263:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961418240:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077179.504288:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077179.504301:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077179/real 1769077179] req@ffff895e453d7480 x1855008961418240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077179.504316:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961418240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077179.504325:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077179.504328:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077179.504332:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961418240:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077179.504339:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961418368:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077179.504357:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077179.504364:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077179/real 1769077179] req@ffff895e453d4e00 x1855008961418368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077179.504372:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961418368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077179.504387:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077179.504389:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077179.504391:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961418368:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077180.015875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077180.207992:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077181.039856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077181.231898:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077182.063914:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077182.255996:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077183.087849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077183.279960:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077184.111852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077184.303912:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077184.623929:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077184.623940:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077184.623945:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077184.623954:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2867 00000100:00080000:3.0:1769077184.623965:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077184.623972:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077184.624041:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961418496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077184.624065:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077184.624068:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077184.624069:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077184.624071:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2867 00000100:00080000:3.0:1769077184.624074:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077184.624076:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077184.624083:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077184.624091:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961418624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077184.624098:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077184.624101:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077184.624102:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077184.624105:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2867 00000100:00100000:0.0:1769077184.624121:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961418496:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077184.624123:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077184.624130:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077184.624135:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077184.624145:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961418752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077184.624182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077184.624214:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077184/real 1769077184] req@ffff895e453d7800 x1855008961418496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077184.624246:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961418496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077184.624256:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077184.624259:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077184.624263:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961418496:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077184.624271:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961418624:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077184.624299:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077184.624314:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077184/real 1769077184] req@ffff895e453d5c00 x1855008961418624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077184.624331:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961418624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077184.624351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077184.624354:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077184.624357:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961418624:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077184.624412:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961418752:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077184.624436:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077184.624445:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077184/real 1769077184] req@ffff895e453d4a80 x1855008961418752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077184.624453:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961418752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077184.624458:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077184.624460:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077184.624462:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961418752:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077185.135898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077185.327893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077186.159851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077186.351913:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077187.183836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077187.375916:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077188.207885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077188.399874:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077189.231882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077189.423972:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077189.743874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077189.743886:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077189.743891:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077189.743917:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2872 00000100:00080000:3.0:1769077189.743928:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077189.743935:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077189.743985:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961418880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077189.744012:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077189.744015:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077189.744016:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077189.744019:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2872 00000100:00080000:3.0:1769077189.744022:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077189.744024:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077189.744034:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077189.744046:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961419008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077189.744054:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077189.744075:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077189.744077:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077189.744081:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2872 00000100:00080000:3.0:1769077189.744084:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077189.744088:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077189.744095:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077189.744098:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961418880:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077189.744106:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961419136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077189.744181:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077189.744217:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077189/real 1769077189] req@ffff895e453d5f80 x1855008961418880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077189.744232:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961418880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077189.744268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077189.744273:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077189.744311:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961418880:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077189.744320:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961419008:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077189.744347:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077189.744361:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077189/real 1769077189] req@ffff895e453d4a80 x1855008961419008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077189.744373:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961419008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077189.744381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077189.744385:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077189.744388:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961419008:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077189.744414:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961419136:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077189.744436:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077189.744446:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077189/real 1769077189] req@ffff895e453d5c00 x1855008961419136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077189.744454:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961419136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077189.744460:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077189.744462:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077189.744465:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961419136:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077190.255880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077190.447899:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077191.279857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077191.471934:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077192.303876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077192.495941:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077193.327869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077193.519917:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077194.351846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077194.543959:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077194.863913:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077194.863925:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077194.863930:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077194.863940:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2877 00000100:00080000:0.0:1769077194.863953:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077194.863961:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077194.864017:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961419264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077194.864044:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077194.864047:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077194.864048:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077194.864052:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2877 00000100:00080000:0.0:1769077194.864057:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077194.864062:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077194.864074:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077194.864082:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961419264:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077194.864093:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961419392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077194.864108:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077194.864112:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077194.864114:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077194.864118:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2877 00000100:00080000:0.0:1769077194.864123:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077194.864146:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077194.864152:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077194.864167:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077194.864169:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961419520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077194.864230:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077194/real 1769077194] req@ffff895e4383d500 x1855008961419264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077194.864247:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961419264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077194.864256:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077194.864260:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077194.864263:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961419264:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077194.864287:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961419392:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077194.864305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077194.864314:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077194/real 1769077194] req@ffff895e4383e300 x1855008961419392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077194.864322:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961419392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077194.864335:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077194.864337:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077194.864340:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961419392:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077194.864344:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961419520:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077194.864363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077194.864374:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077194/real 1769077194] req@ffff895e4383f800 x1855008961419520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077194.864386:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961419520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077194.864395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077194.864400:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077194.864403:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961419520:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077195.375916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077195.567976:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077196.399830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077196.591878:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077197.423815:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077197.615934:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077198.447849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077198.639889:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077199.471888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077199.663977:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077199.983920:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077199.983936:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077199.983942:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077199.983954:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2882 00000100:00080000:0.0:1769077199.983969:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077199.983978:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077199.984031:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961419648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077199.984068:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077199.984072:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077199.984075:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077199.984080:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2882 00000100:00080000:0.0:1769077199.984085:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077199.984090:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077199.984098:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961419648:192.168.204.117@tcp:250:kworker.0 00000008:00100000:0.0:1769077199.984118:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077199.984138:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961419776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077199.984152:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077199.984156:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077199.984178:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077199.984183:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2882 00000100:00080000:0.0:1769077199.984189:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077199.984195:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077199.984202:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077199.984219:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961419904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077199.984244:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961419776:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077199.984271:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961419904:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077200.495862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077200.687933:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077201.519875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077201.711930:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077202.543874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077202.735895:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769077203.056020:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769077203.056093:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077203.056112:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077203.056156:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077203.056166:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769077203.056239:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077199/real 1769077203] req@ffff895e4383ca80 x1855008961419776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077203.056265:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961419776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077203.056285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077203.056291:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077203.056295:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961419776:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077203.056304:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077199/real 1769077203] req@ffff895e4383dc00 x1855008961419904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077203.056318:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961419904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077203.056327:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077203.056331:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077203.056335:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961419904:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077203.056342:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077199/real 1769077203] req@ffff895e4383ea00 x1855008961419648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077203.056355:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961419648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077203.056364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077203.056367:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077203.056392:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961419648:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077203.567910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077203.759961:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077204.591832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077204.783904:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077205.103862:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077205.103872:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077205.103875:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077205.103883:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2887 00000100:00080000:2.0:1769077205.103892:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077205.103898:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077205.103937:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961420032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077205.103959:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077205.103961:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077205.103962:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077205.103965:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2887 00000100:00080000:2.0:1769077205.103968:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077205.103970:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077205.103978:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077205.103987:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961420032:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077205.104006:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961420160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077205.104028:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077205.104032:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077205.104035:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077205.104039:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2887 00000100:00080000:2.0:1769077205.104051:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077205.104056:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:3.0:1769077205.104063:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:2.0:1769077205.104064:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077205.104079:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961420288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077205.104107:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077205/real 1769077205] req@ffff895e44e29180 x1855008961420032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077205.104139:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961420032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077205.104150:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077205.104155:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077205.104159:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961420032:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077205.104187:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961420160:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077205.104214:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077205.104228:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077205/real 1769077205] req@ffff895e44e28a80 x1855008961420160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077205.104241:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961420160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077205.104251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077205.104255:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077205.104258:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961420160:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077205.104265:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961420288:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077205.104302:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077205.104314:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077205/real 1769077205] req@ffff895e44e29500 x1855008961420288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077205.104326:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961420288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077205.104334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077205.104337:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077205.104340:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961420288:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077205.615983:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077205.807905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077206.639811:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077206.831915:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077207.663865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077207.855906:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077208.687825:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077208.879917:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077209.711874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077209.903884:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077210.223899:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077210.223909:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077210.223913:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077210.223923:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2892 00000100:00080000:3.0:1769077210.223960:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077210.223969:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077210.224007:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961420416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077210.224036:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077210.224039:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077210.224040:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077210.224059:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2892 00000100:00080000:3.0:1769077210.224063:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077210.224067:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077210.224075:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077210.224089:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961420416:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077210.224089:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961420544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077210.224104:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077210.224107:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077210.224108:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077210.224112:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2892 00000100:00080000:3.0:1769077210.224114:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077210.224116:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077210.224120:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077210.224129:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961420672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077210.224154:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077210.224189:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077210/real 1769077210] req@ffff895e453d7800 x1855008961420416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077210.224209:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961420416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077210.224220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077210.224225:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077210.224229:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961420416:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077210.224251:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961420544:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077210.224276:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077210.224289:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077210/real 1769077210] req@ffff895e453d4e00 x1855008961420544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077210.224303:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961420544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077210.224336:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077210.224339:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077210.224341:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961420544:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077210.224346:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961420672:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077210.224364:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077210.224372:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077210/real 1769077210] req@ffff895e453d7480 x1855008961420672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077210.224381:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961420672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077210.224386:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077210.224389:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077210.224391:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961420672:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077210.735845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077210.927944:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077211.759944:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077211.951971:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077212.783861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077212.975933:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077213.807858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077213.999898:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077214.831871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077215.023923:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077215.343874:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077215.343900:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077215.343904:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077215.343912:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2897 00000100:00080000:0.0:1769077215.343924:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077215.343930:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077215.343965:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961420800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077215.343992:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077215.344011:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077215.344013:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077215.344017:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2897 00000100:00080000:0.0:1769077215.344022:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077215.344026:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077215.344032:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961420800:192.168.204.117@tcp:250:kworker.0 00000008:00100000:0.0:1769077215.344045:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077215.344066:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961420928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077215.344077:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077215.344080:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077215.344081:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077215.344084:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2897 00000100:00080000:0.0:1769077215.344087:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077215.344090:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077215.344094:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077215.344107:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961421056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077215.344107:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077215.344149:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077215/real 1769077215] req@ffff895e4383ce00 x1855008961420800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077215.344181:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961420800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077215.344190:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077215.344193:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077215.344202:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961420800:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077215.344223:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961420928:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077215.344258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077215.344269:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077215/real 1769077215] req@ffff895e4383c000 x1855008961420928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077215.344279:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961420928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077215.344286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077215.344289:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077215.344291:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961420928:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077215.344296:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961421056:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077215.344308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077215.344315:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077215/real 1769077215] req@ffff895e4383f480 x1855008961421056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077215.344324:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961421056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077215.344330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077215.344332:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077215.344334:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961421056:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077215.855934:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077216.047973:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077216.879863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077217.071954:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077217.903873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077218.095891:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077218.927901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077219.119911:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077219.951893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077220.144003:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077220.463873:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077220.463883:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077220.463886:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077220.463895:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2903 00000100:00080000:0.0:1769077220.463905:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077220.463912:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077220.463945:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961421184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077220.463987:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077220.463990:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077220.463991:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077220.463995:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2903 00000100:00080000:0.0:1769077220.463998:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077220.464002:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077220.464010:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077220.464022:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961421312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077220.464030:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077220.464033:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077220.464034:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077220.464053:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2903 00000100:00080000:0.0:1769077220.464055:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077220.464061:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077220.464063:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961421184:192.168.204.117@tcp:250:kworker.0 00000008:00100000:0.0:1769077220.464065:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077220.464076:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961421440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077220.464154:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077220.464201:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077220/real 1769077220] req@ffff895e4383c380 x1855008961421184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077220.464225:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961421184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077220.464237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077220.464242:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077220.464247:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961421184:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077220.464257:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961421312:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077220.464284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077220.464299:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077220/real 1769077220] req@ffff895e4383f100 x1855008961421312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077220.464316:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961421312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077220.464328:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077220.464332:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077220.464337:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961421312:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077220.464363:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961421440:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077220.464387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077220.464400:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077220/real 1769077220] req@ffff895e4383fb80 x1855008961421440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077220.464433:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961421440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077220.464461:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077220.464464:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077220.464468:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961421440:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077220.975901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077221.167927:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077221.999842:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077222.191883:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077223.023862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077223.215915:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077224.047807:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077224.239902:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077225.071849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077225.263888:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077225.583871:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077225.583880:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077225.583883:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077225.583891:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2908 00000100:00080000:0.0:1769077225.583916:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077225.583922:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077225.583952:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961421568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077225.583977:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077225.583979:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077225.583981:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077225.583984:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2908 00000100:00080000:0.0:1769077225.583987:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077225.583991:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077225.584000:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077225.584013:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961421696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077225.584027:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077225.584028:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961421568:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077225.584041:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077225.584043:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077225.584049:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2908 00000100:00080000:0.0:1769077225.584052:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077225.584056:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077225.584061:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077225.584074:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961421824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077225.584096:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077225.584135:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077225/real 1769077225] req@ffff895e4383c700 x1855008961421568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077225.584154:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961421568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077225.584165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077225.584169:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077225.584173:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961421568:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077225.584195:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961421696:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077225.584217:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077225.584228:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077225/real 1769077225] req@ffff895e4383e680 x1855008961421696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077225.584252:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961421696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077225.584274:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077225.584277:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077225.584281:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961421696:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077225.584288:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961421824:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077225.584310:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077225.584322:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077225/real 1769077225] req@ffff895e4383d880 x1855008961421824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077225.584335:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961421824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077225.584342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077225.584346:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077225.584350:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961421824:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077226.095824:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077226.287902:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077227.119886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077227.311985:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077228.143839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077228.335912:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077229.167869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077229.359933:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077230.191837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077230.383966:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077230.703928:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077230.703940:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077230.703945:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077230.703955:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2913 00000100:00080000:3.0:1769077230.703968:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077230.703990:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077230.704026:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961421952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077230.704074:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077230.704094:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077230.704098:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077230.704104:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2913 00000100:00100000:0.0:1769077230.704110:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961421952:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077230.704113:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077230.704139:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077230.704155:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077230.704176:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961422080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077230.704206:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077230.704210:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077230.704212:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077230.704216:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2913 00000100:00080000:3.0:1769077230.704220:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077230.704221:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769077230.704227:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077230.704233:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077230.704246:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961422208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077230.704303:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077230/real 1769077230] req@ffff895e453d5500 x1855008961421952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077230.704339:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961421952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077230.704347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077230.704350:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077230.704353:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961421952:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077230.704374:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961422080:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077230.704399:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077230.704432:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077230/real 1769077230] req@ffff895e453d7b80 x1855008961422080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077230.704448:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961422080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077230.704456:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077230.704459:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077230.704462:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961422080:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077230.704467:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961422208:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077230.704502:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077230.704510:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077230/real 1769077230] req@ffff895e453d7100 x1855008961422208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077230.704531:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961422208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077230.704537:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077230.704539:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077230.704542:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961422208:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077231.215846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077231.407914:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077232.239847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077232.431869:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077233.263879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077233.455882:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077234.287883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077234.479882:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077235.311897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077235.503862:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077235.823898:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077235.823909:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077235.823914:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077235.823922:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2918 00000100:00080000:0.0:1769077235.823932:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077235.823938:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077235.823971:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961422336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077235.823995:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077235.823997:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077235.823998:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077235.824001:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2918 00000100:00080000:0.0:1769077235.824004:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077235.824006:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077235.824014:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077235.824027:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961422336:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077235.824045:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961422464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077235.824054:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077235.824057:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077235.824059:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077235.824062:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2918 00000100:00080000:0.0:1769077235.824066:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077235.824069:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077235.824072:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077235.824080:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961422592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077235.824092:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077235.824133:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077235/real 1769077235] req@ffff895e4383ed80 x1855008961422336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077235.824149:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961422336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077235.824163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077235.824167:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077235.824170:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961422336:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077235.824190:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961422464:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077235.824210:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077235.824223:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077235/real 1769077235] req@ffff895e4383d180 x1855008961422464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077235.824237:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961422464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077235.824247:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077235.824251:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077235.824265:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961422464:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077235.824273:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961422592:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077235.824291:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077235.824300:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077235/real 1769077235] req@ffff895e4383df80 x1855008961422592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077235.824308:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961422592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077235.824314:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077235.824316:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077235.824318:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961422592:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077236.335870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077236.527930:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077237.359841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077237.551872:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077238.383877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077238.575913:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077239.407898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077239.599955:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077240.431804:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077240.623905:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077240.943926:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077240.943937:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077240.943942:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077240.943951:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2923 00000100:00080000:1.0:1769077240.943963:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077240.943970:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077240.944007:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961422720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077240.944048:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077240.944052:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077240.944055:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077240.944059:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2923 00000100:00080000:1.0:1769077240.944063:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077240.944069:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077240.944079:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077240.944088:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961422720:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077240.944098:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961422848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077240.944118:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077240.944122:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077240.944138:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077240.944152:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2923 00000800:00000100:3.0:1769077240.944157:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:1.0:1769077240.944160:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077240.944167:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077240.944175:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077240.944198:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961422976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077240.944217:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077240/real 1769077240] req@ffff895e4713ea00 x1855008961422720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077240.944234:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961422720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077240.944241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077240.944245:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077240.944248:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961422720:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077240.944266:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961422848:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077240.944293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077240.944301:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077240/real 1769077240] req@ffff895e4713f100 x1855008961422848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077240.944309:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961422848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077240.944315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077240.944317:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077240.944320:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961422848:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077240.944330:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961422976:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077240.944344:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077240.944359:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077240/real 1769077240] req@ffff895e4713e300 x1855008961422976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077240.944367:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961422976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077240.944372:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077240.944374:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077240.944375:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961422976:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077241.455826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077241.647884:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077242.479825:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077242.671897:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077243.503811:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077243.695854:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077244.527870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077244.719922:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077245.551822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077245.743887:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077246.063898:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077246.063906:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077246.063909:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077246.063917:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2928 00000100:00080000:2.0:1769077246.063927:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077246.063932:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077246.063978:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961423104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077246.064007:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077246.064012:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077246.064014:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077246.064018:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2928 00000100:00080000:2.0:1769077246.064023:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077246.064028:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077246.064038:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077246.064051:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961423104:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077246.064053:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961423232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077246.064068:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077246.064071:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077246.064073:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077246.064077:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2928 00000100:00080000:2.0:1769077246.064080:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077246.064085:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077246.064090:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077246.064100:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961423360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077246.064115:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077246.064156:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077246/real 1769077246] req@ffff895e44e28e00 x1855008961423104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077246.064172:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961423104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077246.064180:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077246.064183:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077246.064186:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961423104:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077246.064219:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961423232:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077246.064246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077246.064258:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077246/real 1769077246] req@ffff895e44e2b480 x1855008961423232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077246.064267:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961423232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077246.064273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077246.064275:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077246.064278:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961423232:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077246.064282:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961423360:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077246.064294:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077246.064300:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077246/real 1769077246] req@ffff895e44e29f80 x1855008961423360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077246.064308:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961423360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077246.064312:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077246.064314:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077246.064316:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961423360:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077246.575900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077246.767906:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077247.599807:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077247.791934:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077248.623924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077248.815898:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077249.647862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077249.839988:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077250.671908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077250.863943:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077251.183898:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077251.183935:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077251.183940:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077251.183950:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2933 00000100:00080000:3.0:1769077251.183963:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077251.183970:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077251.184009:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961423488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077251.184043:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077251.184047:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077251.184049:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077251.184054:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2933 00000100:00080000:3.0:1769077251.184058:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077251.184072:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077251.184083:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077251.184099:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961423616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077251.184108:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077251.184111:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077251.184113:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077251.184118:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961423488:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077251.184119:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2933 00000100:00080000:3.0:1769077251.184123:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077251.184130:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077251.184135:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077251.184144:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961423744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077251.184216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077251.184285:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077251/real 1769077251] req@ffff895e453d4700 x1855008961423488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077251.184309:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961423488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077251.184320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077251.184325:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077251.184330:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961423488:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077251.184368:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961423616:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077251.184398:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077251.184414:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077251/real 1769077251] req@ffff895e453d5880 x1855008961423616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077251.184429:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961423616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077251.184438:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077251.184441:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077251.184445:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961423616:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077251.184452:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961423744:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077251.184484:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077251.184497:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077251/real 1769077251] req@ffff895e453d5180 x1855008961423744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077251.184510:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961423744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077251.184519:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077251.184523:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077251.184526:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961423744:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077251.695962:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077251.887969:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077252.719855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077252.911873:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077253.743842:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077253.935976:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077254.767832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077254.959888:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077255.791856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077255.983942:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077256.303828:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077256.303837:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077256.303842:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077256.303852:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2938 00000100:00080000:1.0:1769077256.303863:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077256.303914:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077256.303947:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961423872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077256.303969:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077256.303973:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077256.303975:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077256.303979:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2938 00000100:00080000:1.0:1769077256.303983:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077256.304001:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077256.304010:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077256.304015:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961423872:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077256.304023:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961424000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077256.304033:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077256.304036:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077256.304038:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077256.304041:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2938 00000100:00080000:1.0:1769077256.304045:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077256.304048:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077256.304052:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077256.304061:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961424128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077256.304073:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077256.304109:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077256/real 1769077256] req@ffff895e4713d880 x1855008961423872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077256.304125:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961423872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077256.304132:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077256.304136:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077256.304139:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961423872:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077256.304178:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961424000:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077256.304197:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077256.304205:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077256/real 1769077256] req@ffff895e4713ed80 x1855008961424000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077256.304213:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961424000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077256.304219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077256.304221:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077256.304223:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961424000:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077256.304227:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961424128:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077256.304237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077256.304243:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077256/real 1769077256] req@ffff895e4713c000 x1855008961424128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077256.304256:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961424128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077256.304262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077256.304264:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077256.304266:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961424128:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077256.815857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077257.007898:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077257.839888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077258.031903:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077258.863853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077259.055881:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077259.887859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077260.080010:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077260.911847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077261.103933:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077261.423885:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077261.423895:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077261.423899:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077261.423909:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2943 00000100:00080000:2.0:1769077261.423920:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077261.423927:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077261.423962:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961424256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077261.423992:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077261.423996:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077261.423998:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077261.424002:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2943 00000100:00080000:2.0:1769077261.424005:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077261.424010:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077261.424019:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077261.424031:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961424384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077261.424040:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961424256:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077261.424047:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077261.424051:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077261.424067:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077261.424099:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2943 00000100:00080000:2.0:1769077261.424107:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077261.424111:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:0.0:1769077261.424117:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:2.0:1769077261.424118:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077261.424134:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961424512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077261.424153:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077261/real 1769077261] req@ffff895e44e2a300 x1855008961424256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077261.424168:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961424256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077261.424183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077261.424186:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077261.424189:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961424256:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077261.424210:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961424384:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077261.424226:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077261.424234:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077261/real 1769077261] req@ffff895e44e2aa00 x1855008961424384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077261.424249:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961424384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077261.424256:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077261.424258:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077261.424260:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961424384:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077261.424264:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961424512:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077261.424277:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077261.424283:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077261/real 1769077261] req@ffff895e44e2b100 x1855008961424512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077261.424290:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961424512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077261.424295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077261.424297:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077261.424299:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961424512:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077261.935853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077262.127970:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077262.959825:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077263.151926:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077263.983841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077264.175926:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077265.007860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077265.199970:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077266.031850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077266.223903:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077266.543883:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077266.543892:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077266.543897:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077266.543921:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2949 00000100:00080000:0.0:1769077266.543941:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077266.543952:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077266.544014:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961424640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077266.544037:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077266.544041:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077266.544042:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077266.544046:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2949 00000100:00080000:0.0:1769077266.544050:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077266.544053:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077266.544061:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077266.544073:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961424640:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077266.544074:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961424768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077266.544088:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077266.544090:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077266.544091:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077266.544094:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2949 00000100:00080000:0.0:1769077266.544097:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077266.544100:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077266.544102:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077266.544112:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961424896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077266.544168:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961424768:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077266.544195:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961424896:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077267.055871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077267.247936:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077268.079892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077268.272000:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077269.104049:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077269.296053:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769077269.616210:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769077269.616264:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077269.616285:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077269.616362:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077269.616382:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769077269.616450:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077266/real 1769077269] req@ffff895e4383d180 x1855008961424768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077269.616497:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961424768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077269.616530:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077269.616539:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077269.616548:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961424768:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077269.616561:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077266/real 1769077269] req@ffff895e4383ed80 x1855008961424896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077269.616596:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961424896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077269.616611:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077269.616617:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077269.616624:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961424896:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077269.616637:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077266/real 1769077269] req@ffff895e4383df80 x1855008961424640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077269.616693:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961424640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077269.616709:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077269.616715:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077269.616721:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961424640:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077270.127885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077270.320006:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077271.151948:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077271.344002:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077271.663923:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077271.663934:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077271.664033:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077271.664044:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2954 00000100:00080000:1.0:1769077271.664056:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077271.664063:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077271.664102:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961425024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077271.664133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077271.664137:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077271.664140:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077271.664144:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2954 00000100:00080000:1.0:1769077271.664149:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077271.664154:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077271.664165:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077271.664179:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961425152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077271.664186:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961425024:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077271.664196:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077271.664202:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077271.664204:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077271.664210:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2954 00000100:00080000:1.0:1769077271.664216:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077271.664221:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077271.664227:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077271.664241:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961425280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077271.664282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077271.664371:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077271/real 1769077271] req@ffff895e4713c700 x1855008961425024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077271.664395:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961425024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077271.664408:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077271.664413:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077271.664418:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961425024:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077271.664447:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961425152:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077271.664475:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077271.664487:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077271/real 1769077271] req@ffff895e4713fb80 x1855008961425152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077271.664500:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961425152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077271.664521:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077271.664525:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077271.664528:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961425152:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077271.664534:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961425280:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077271.664556:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077271.664568:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077271/real 1769077271] req@ffff895e4713c000 x1855008961425280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077271.664587:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961425280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077271.664595:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077271.664599:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077271.664602:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961425280:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077272.175851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077272.367860:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077273.199869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077273.392008:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077274.223873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077274.415951:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077275.248022:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077275.439992:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077276.271900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077276.463924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077276.783898:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077276.783907:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077276.783912:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077276.783922:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2959 00000100:00080000:3.0:1769077276.783951:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077276.783962:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077276.784024:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961425408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077276.784077:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077276.784081:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077276.784083:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077276.784087:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2959 00000100:00080000:3.0:1769077276.784092:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077276.784096:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077276.784105:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077276.784120:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961425536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077276.784130:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077276.784134:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077276.784135:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077276.784138:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2959 00000100:00080000:3.0:1769077276.784142:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077276.784145:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077276.784150:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077276.784175:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961425408:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077276.784178:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961425664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077276.784266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077276.784300:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077276/real 1769077276] req@ffff895e453d6680 x1855008961425408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077276.784315:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961425408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077276.784323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077276.784327:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077276.784330:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961425408:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077276.784335:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961425536:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077276.784361:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077276.784378:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077276/real 1769077276] req@ffff895e453d4380 x1855008961425536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077276.784392:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961425536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077276.784401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077276.784405:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077276.784409:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961425536:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077276.784450:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961425664:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077276.784490:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077276.784516:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077276/real 1769077276] req@ffff895e453d6d80 x1855008961425664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077276.784524:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961425664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077276.784533:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077276.784536:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077276.784539:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961425664:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077277.295912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077277.488017:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077278.319892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077278.511943:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077279.343922:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077279.536025:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077280.367847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077280.559969:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077281.391887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077281.584018:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077281.903856:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077281.903866:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077281.903869:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077281.903877:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2964 00000100:00080000:3.0:1769077281.903902:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077281.903908:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077281.903936:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961425792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077281.903961:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077281.903963:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077281.903964:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077281.903967:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2964 00000100:00080000:3.0:1769077281.903976:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077281.903978:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077281.903985:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077281.903995:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961425920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077281.904003:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077281.904005:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077281.904006:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077281.904009:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2964 00000100:00080000:3.0:1769077281.904010:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077281.904013:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077281.904016:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077281.904026:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961426048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077281.904092:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961425792:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769077281.904228:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077281.904296:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077281/real 1769077281] req@ffff895e453d6300 x1855008961425792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077281.904318:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961425792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077281.904331:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077281.904337:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077281.904343:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961425792:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077281.904353:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961425920:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077281.904378:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077281.904390:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077281/real 1769077281] req@ffff895e453d6d80 x1855008961425920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077281.904447:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961425920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077281.904457:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077281.904461:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077281.904465:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961425920:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077281.904472:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961426048:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077281.904495:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077281.904507:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077281/real 1769077281] req@ffff895e453d4380 x1855008961426048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077281.904548:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961426048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077281.904563:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077281.904570:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077281.904577:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961426048:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077282.415826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077282.607931:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077283.439844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077283.631891:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077284.463832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077284.655946:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077285.487873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077285.679948:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077286.511833:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077286.703946:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077287.023838:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077287.023864:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077287.023868:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077287.023876:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2969 00000100:00080000:3.0:1769077287.023886:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077287.023893:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077287.023922:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961426176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077287.023945:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077287.023948:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077287.023949:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077287.023952:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2969 00000100:00080000:3.0:1769077287.023954:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077287.023956:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077287.023964:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077287.023974:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961426304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077287.023982:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077287.023984:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077287.023986:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077287.023988:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2969 00000100:00080000:3.0:1769077287.023991:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077287.023997:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077287.023998:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961426176:192.168.204.117@tcp:250:kworker.0 00000008:00100000:3.0:1769077287.024003:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077287.024018:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961426432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077287.024061:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077287.024095:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077287/real 1769077287] req@ffff895e453d6680 x1855008961426176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077287.024108:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961426176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077287.024126:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077287.024130:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077287.024132:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961426176:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077287.024138:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961426304:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077287.024154:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077287.024162:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077287/real 1769077287] req@ffff895e453d5180 x1855008961426304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077287.024169:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961426304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077287.024175:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077287.024177:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077287.024179:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961426304:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077287.024197:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961426432:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077287.024208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077287.024213:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077287/real 1769077287] req@ffff895e453d5880 x1855008961426432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077287.024220:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961426432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077287.024225:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077287.024227:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077287.024229:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961426432:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077287.535830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077287.727961:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077288.559921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077288.751897:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077289.583883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077289.775918:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077290.607879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077290.799942:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077291.631881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077291.823987:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077292.143968:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077292.143987:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077292.143992:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077292.144006:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2974 00000100:00080000:2.0:1769077292.144021:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077292.144030:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077292.144084:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961426560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077292.144120:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077292.144124:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077292.144126:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077292.144131:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2974 00000100:00080000:2.0:1769077292.144134:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077292.144138:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077292.144151:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077292.144167:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961426688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077292.144195:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077292.144199:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077292.144200:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077292.144204:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2974 00000100:00080000:2.0:1769077292.144207:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077292.144213:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077292.144216:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961426560:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769077292.144219:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077292.144256:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961426816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077292.144344:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077292.144426:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077292/real 1769077292] req@ffff895e44e28000 x1855008961426560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077292.144448:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961426560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077292.144460:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077292.144465:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077292.144480:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961426560:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077292.144488:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961426688:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077292.144515:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077292.144526:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077292/real 1769077292] req@ffff895e44e28380 x1855008961426688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077292.144539:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961426688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077292.144547:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077292.144550:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077292.144553:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961426688:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077292.144653:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961426816:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077292.144680:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077292.144692:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077292/real 1769077292] req@ffff895e44e29880 x1855008961426816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077292.144721:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961426816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077292.144729:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077292.144733:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077292.144736:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961426816:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077292.655888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077292.847998:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077293.679857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077293.871911:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077294.703885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077294.895858:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077295.727853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077295.919897:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077296.751825:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077296.943913:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077297.263941:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077297.263956:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077297.263961:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077297.263975:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2979 00000100:00080000:2.0:1769077297.263989:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077297.263999:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077297.264049:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961426944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077297.264083:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077297.264087:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077297.264089:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077297.264094:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2979 00000100:00080000:2.0:1769077297.264098:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077297.264103:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077297.264114:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077297.264130:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961427072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077297.264143:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961426944:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077297.264155:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077297.264181:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077297.264183:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077297.264192:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2979 00000100:00080000:2.0:1769077297.264197:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077297.264202:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077297.264210:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077297.264225:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961427200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077297.264236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077297.264281:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077297/real 1769077297] req@ffff895e44e29880 x1855008961426944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077297.264303:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961426944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077297.264312:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077297.264316:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077297.264319:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961426944:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077297.264353:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961427072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077297.264372:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077297.264381:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077297/real 1769077297] req@ffff895e44e28380 x1855008961427072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077297.264389:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961427072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077297.264395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077297.264398:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077297.264400:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961427072:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077297.264404:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961427200:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077297.264415:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077297.264421:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077297/real 1769077297] req@ffff895e44e28000 x1855008961427200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077297.264428:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961427200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077297.264433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077297.264435:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077297.264437:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961427200:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077297.775891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077297.967870:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077298.799862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077298.991897:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077299.823846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077300.015909:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077300.847891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077301.039916:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077301.871836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077302.063919:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077302.383858:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077302.383868:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077302.383872:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077302.383880:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2984 00000100:00080000:2.0:1769077302.383891:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077302.383897:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077302.383928:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961427328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077302.383952:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077302.383955:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077302.383956:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077302.383959:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2984 00000100:00080000:2.0:1769077302.383961:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077302.383964:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077302.383995:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077302.384005:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961427456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077302.384013:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077302.384016:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077302.384017:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077302.384021:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2984 00000100:00080000:2.0:1769077302.384024:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077302.384026:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077302.384030:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077302.384041:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961427584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077302.384051:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961427328:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769077302.384118:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077302.384167:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077302/real 1769077302] req@ffff895e44e2b100 x1855008961427328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077302.384189:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961427328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077302.384197:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077302.384200:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077302.384203:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961427328:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077302.384209:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961427456:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077302.384227:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077302.384235:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077302/real 1769077302] req@ffff895e44e2aa00 x1855008961427456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077302.384243:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961427456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077302.384248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077302.384250:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077302.384252:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961427456:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077302.384272:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961427584:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077302.384282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077302.384287:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077302/real 1769077302] req@ffff895e44e2a300 x1855008961427584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077302.384295:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961427584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077302.384309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077302.384311:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077302.384313:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961427584:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077302.895829:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077303.087911:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077303.919834:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077304.111952:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077304.943849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077305.135892:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077305.967956:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077306.159967:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077306.991879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077307.183941:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077307.503855:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077307.503866:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077307.503871:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077307.503881:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2990 00000100:00080000:2.0:1769077307.503893:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077307.503900:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077307.503937:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961427712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077307.503970:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077307.503974:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077307.503988:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077307.503993:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2990 00000100:00080000:2.0:1769077307.503998:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077307.504003:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077307.504013:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077307.504027:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961427712:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077307.504031:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961427840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077307.504047:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077307.504050:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077307.504053:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077307.504057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2990 00000100:00080000:2.0:1769077307.504061:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077307.504065:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077307.504071:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077307.504082:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961427968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077307.504100:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077307.504144:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077307/real 1769077307] req@ffff895e44e29f80 x1855008961427712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077307.504164:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961427712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077307.504175:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077307.504178:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077307.504181:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961427712:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077307.504201:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961427840:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077307.504220:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077307.504229:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077307/real 1769077307] req@ffff895e44e2b480 x1855008961427840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077307.504237:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961427840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077307.504243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077307.504245:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077307.504248:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961427840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077307.504259:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961427968:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077307.504277:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077307.504285:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077307/real 1769077307] req@ffff895e44e28e00 x1855008961427968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077307.504292:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961427968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077307.504321:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077307.504324:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077307.504326:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961427968:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077308.015856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077308.207953:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077309.039899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077309.231951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077310.063892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077310.256052:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077311.087874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077311.279936:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077312.111851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077312.303928:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077312.623854:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077312.623893:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077312.623898:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077312.623911:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 2995 00000100:00080000:1.0:1769077312.623925:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077312.623933:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077312.623985:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961428096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077312.624021:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077312.624025:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077312.624027:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077312.624037:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2995 00000100:00080000:1.0:1769077312.624041:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077312.624045:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077312.624056:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077312.624070:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961428224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077312.624082:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077312.624091:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077312.624094:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077312.624099:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 2995 00000100:00100000:0.0:1769077312.624100:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961428096:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077312.624105:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077312.624114:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077312.624121:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077312.624138:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961428352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077312.624173:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077312.624233:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077312/real 1769077312] req@ffff895e4713ed80 x1855008961428096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077312.624250:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961428096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077312.624262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077312.624267:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077312.624272:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961428096:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077312.624299:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961428224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077312.624319:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077312.624328:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077312/real 1769077312] req@ffff895e4713d880 x1855008961428224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077312.624351:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961428224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077312.624365:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077312.624368:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077312.624370:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961428224:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077312.624375:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961428352:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077312.624390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077312.624397:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077312/real 1769077312] req@ffff895e4713e300 x1855008961428352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077312.624405:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961428352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077312.624410:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077312.624412:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077312.624414:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961428352:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077313.135845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077313.327869:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077314.159853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077314.351935:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077315.183871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077315.375874:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077316.207880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077316.399873:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077317.231830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077317.423853:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077317.743861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077317.743869:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077317.743873:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077317.743880:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3000 00000100:00080000:1.0:1769077317.743889:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077317.743895:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077317.743924:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961428480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077317.743947:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077317.743950:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077317.743951:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077317.743953:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3000 00000100:00080000:1.0:1769077317.743960:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077317.743962:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077317.743969:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077317.743978:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961428608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077317.743987:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077317.743990:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077317.743992:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077317.744004:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961428480:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077317.744016:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3000 00000100:00080000:1.0:1769077317.744021:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077317.744025:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077317.744031:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077317.744044:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961428736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077317.744067:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077317.744102:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077317/real 1769077317] req@ffff895e4713f100 x1855008961428480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077317.744155:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961428480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077317.744178:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077317.744182:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077317.744186:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961428480:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077317.744193:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961428608:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077317.744216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077317.744227:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077317/real 1769077317] req@ffff895e4713ea00 x1855008961428608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077317.744239:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961428608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077317.744247:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077317.744250:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077317.744253:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961428608:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077317.744275:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961428736:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077317.744290:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077317.744299:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077317/real 1769077317] req@ffff895e4713d500 x1855008961428736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077317.744309:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961428736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077317.744317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077317.744320:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077317.744323:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961428736:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077318.255841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077318.447882:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077319.279809:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077319.471863:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077320.303837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077320.495852:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077321.327804:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077321.519907:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077322.351840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077322.543920:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077322.863876:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077322.863883:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077322.863887:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077322.863894:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3005 00000100:00080000:0.0:1769077322.863902:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077322.863908:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077322.863965:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961428864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077322.863994:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077322.863998:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077322.863999:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077322.864003:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3005 00000100:00080000:0.0:1769077322.864007:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077322.864011:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077322.864020:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077322.864031:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961428992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077322.864051:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077322.864054:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077322.864055:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077322.864059:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3005 00000100:00100000:3.0:1769077322.864059:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961428864:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077322.864062:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077322.864065:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077322.864084:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077322.864094:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961429120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077322.864114:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077322.864158:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077322/real 1769077322] req@ffff895e4383d880 x1855008961428864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077322.864197:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961428864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077322.864203:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077322.864206:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077322.864208:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961428864:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077322.864226:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961428992:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077322.864242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077322.864250:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077322/real 1769077322] req@ffff895e4383e680 x1855008961428992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077322.864257:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961428992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077322.864262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077322.864264:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077322.864265:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961428992:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077322.864268:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961429120:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077322.864289:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077322.864298:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077322/real 1769077322] req@ffff895e4383c700 x1855008961429120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077322.864309:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961429120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077322.864327:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077322.864331:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077322.864334:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961429120:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077323.375834:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077323.567896:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077324.399836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077324.591899:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077325.423939:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077325.616008:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077326.447852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077326.639890:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077327.471825:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077327.663903:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077327.983855:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077327.983865:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077327.983870:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077327.983878:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3010 00000100:00080000:0.0:1769077327.983888:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077327.983894:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077327.983928:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961429248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077327.983988:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077327.983993:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077327.983995:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077327.983999:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961429248:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077327.984002:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3010 00000100:00080000:0.0:1769077327.984008:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077327.984012:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077327.984020:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077327.984035:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961429376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077327.984045:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077327.984048:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077327.984050:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769077327.984057:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769077327.984063:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3010 00000100:00080000:0.0:1769077327.984082:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077327.984087:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077327.984091:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077327/real 1769077327] req@ffff895e4383fb80 x1855008961429248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1769077327.984095:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077327.984108:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961429248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077327.984113:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961429504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077327.984118:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077327.984125:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077327.984130:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961429248:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077327.984149:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961429376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077327.984163:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077327.984169:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077327/real 1769077327] req@ffff895e4383f100 x1855008961429376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077327.984176:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961429376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077327.984187:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077327.984189:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077327.984191:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961429376:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077327.984195:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961429504:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077327.984206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077327.984212:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077327/real 1769077327] req@ffff895e4383c380 x1855008961429504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077327.984218:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961429504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077327.984222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077327.984224:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077327.984225:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961429504:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077328.495834:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077328.687854:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077329.519854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077329.711944:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077330.543845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077330.735897:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077331.567839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077331.759951:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077332.591852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077332.784011:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077333.103854:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077333.103863:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077333.103879:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077333.103889:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3015 00000100:00080000:0.0:1769077333.103899:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077333.103905:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077333.103938:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961429632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077333.103959:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077333.103962:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077333.103963:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077333.103966:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3015 00000100:00080000:0.0:1769077333.103968:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077333.103971:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077333.103977:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077333.103985:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961429760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077333.103993:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077333.104006:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961429632:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077333.104008:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077333.104011:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077333.104016:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3015 00000100:00080000:0.0:1769077333.104019:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077333.104022:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077333.104026:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077333.104036:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961429888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077333.104094:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961429760:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077333.104121:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961429888:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077333.615854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077333.807932:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077334.639835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077334.831882:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077335.663868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077335.855928:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769077336.176051:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769077336.176093:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077336.176107:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769077336.176142:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769077336.176151:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769077336.176335:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077333/real 1769077336] req@ffff895e4383ce00 x1855008961429888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077336.176367:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961429888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077336.176382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077336.176387:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077336.176392:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961429888:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077336.176402:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077333/real 1769077336] req@ffff895e4383f480 x1855008961429632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077336.176415:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961429632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077336.176422:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077336.176426:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077336.176429:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961429632:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077336.176437:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077333/real 1769077336] req@ffff895e4383c000 x1855008961429760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077336.176449:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961429760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077336.176459:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077336.176462:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077336.176485:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961429760:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077336.687910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077336.880049:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077337.711843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077337.903905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077338.223886:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077338.223898:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077338.223902:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077338.223913:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3020 00000100:00080000:1.0:1769077338.223924:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077338.223932:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077338.223968:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961430016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077338.223999:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077338.224020:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077338.224023:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077338.224028:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3020 00000100:00100000:3.0:1769077338.224034:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961430016:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077338.224035:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077338.224045:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077338.224055:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077338.224073:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961430144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077338.224085:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077338.224089:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077338.224091:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077338.224095:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3020 00000100:00080000:1.0:1769077338.224100:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077338.224106:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:3.0:1769077338.224108:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:1.0:1769077338.224114:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077338.224128:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961430272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077338.224152:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077338/real 1769077338] req@ffff895e4713ce00 x1855008961430016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077338.224174:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961430016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077338.224185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077338.224190:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077338.224194:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961430016:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077338.224220:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961430144:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077338.224245:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077338.224259:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077338/real 1769077338] req@ffff895e4713f480 x1855008961430144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077338.224272:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961430144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077338.224281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077338.224285:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077338.224288:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961430144:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077338.224295:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961430272:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077338.224324:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077338.224338:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077338/real 1769077338] req@ffff895e4713ca80 x1855008961430272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077338.224350:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961430272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077338.224357:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077338.224361:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077338.224363:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961430272:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077338.735842:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077338.927901:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077339.759928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077339.951921:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077340.783850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077340.975979:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077341.807916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077341.999954:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077342.831916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077343.024016:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077343.343890:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077343.343905:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077343.343912:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077343.343927:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3025 00000100:00080000:1.0:1769077343.343944:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077343.343955:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077343.344037:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961430400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077343.344082:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077343.344087:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077343.344090:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077343.344095:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3025 00000100:00080000:1.0:1769077343.344101:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077343.344106:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077343.344118:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077343.344132:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961430528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077343.344147:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077343.344152:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077343.344155:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077343.344159:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3025 00000100:00080000:1.0:1769077343.344163:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077343.344169:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077343.344174:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077343.344194:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961430656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077343.344206:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961430400:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769077343.344337:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077343.344400:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077343/real 1769077343] req@ffff895e4713f800 x1855008961430400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077343.344427:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961430400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077343.344442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077343.344448:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077343.344454:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961430400:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077343.344464:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961430528:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077343.344511:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077343.344527:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077343/real 1769077343] req@ffff895e4713dc00 x1855008961430528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077343.344543:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961430528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077343.344558:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077343.344565:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077343.344572:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961430528:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077343.344622:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961430656:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077343.344666:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077343.344692:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077343/real 1769077343] req@ffff895e4713d180 x1855008961430656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077343.344812:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961430656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077343.344823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077343.344827:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077343.344832:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961430656:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077343.855857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077344.047966:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077344.879826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077345.071949:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077345.903877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077346.095959:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077346.927921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077347.119998:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077347.951893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077348.143934:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077348.463924:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077348.463937:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077348.463941:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077348.463952:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3031 00000100:00080000:0.0:1769077348.463964:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077348.463971:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077348.464025:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961430784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077348.464051:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077348.464054:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077348.464055:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077348.464058:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3031 00000100:00080000:0.0:1769077348.464061:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077348.464063:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077348.464071:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077348.464080:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961430912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077348.464108:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077348.464110:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077348.464113:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077348.464116:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961430784:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077348.464118:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3031 00000100:00080000:0.0:1769077348.464121:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077348.464125:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077348.464132:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077348.464142:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961431040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077348.464210:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077348.464263:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077348/real 1769077348] req@ffff895e4383ea00 x1855008961430784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077348.464281:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961430784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077348.464290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077348.464295:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077348.464298:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961430784:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077348.464322:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961430912:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077348.464340:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077348.464350:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077348/real 1769077348] req@ffff895e4383dc00 x1855008961430912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077348.464358:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961430912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077348.464364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077348.464367:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077348.464369:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961430912:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077348.464373:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961431040:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077348.464390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077348.464397:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077348/real 1769077348] req@ffff895e4383ca80 x1855008961431040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077348.464415:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961431040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077348.464421:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077348.464423:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077348.464426:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961431040:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077348.975884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077349.167967:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077349.999842:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077350.191879:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077351.023890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077351.215968:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077352.047865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077352.239875:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077353.071853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077353.263937:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077353.583826:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077353.583836:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077353.583840:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077353.583850:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3036 00000100:00080000:3.0:1769077353.583861:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077353.583868:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077353.583907:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961431168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077353.583940:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077353.583944:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077353.583946:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077353.583950:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3036 00000100:00080000:3.0:1769077353.583955:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077353.583974:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077353.583982:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961431168:192.168.204.117@tcp:250:kworker.0 00000008:00100000:3.0:1769077353.583983:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077353.584009:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961431296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077353.584023:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077353.584026:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077353.584029:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077353.584033:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3036 00000100:00080000:3.0:1769077353.584039:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077353.584044:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769077353.584045:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077353.584054:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077353.584075:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961431424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077353.584082:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077353/real 1769077353] req@ffff895e453d4700 x1855008961431168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077353.584098:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961431168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077353.584107:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077353.584110:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077353.584114:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961431168:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077353.584132:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961431296:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077353.584147:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077353.584154:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077353/real 1769077353] req@ffff895e453d7100 x1855008961431296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077353.584170:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961431296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077353.584176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077353.584178:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077353.584180:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961431296:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077353.584183:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961431424:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077353.584196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077353.584202:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077353/real 1769077353] req@ffff895e453d7b80 x1855008961431424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077353.584209:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961431424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077353.584214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077353.584216:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077353.584218:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961431424:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077354.095891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077354.287895:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077355.119879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077355.311936:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077356.143899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077356.335921:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077357.167886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077357.359897:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077358.191846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077358.384062:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077358.703853:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077358.703865:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077358.703869:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077358.703880:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3041 00000100:00080000:3.0:1769077358.703892:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077358.703899:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077358.703942:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961431552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077358.703989:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077358.703992:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077358.703993:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077358.703997:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3041 00000100:00080000:3.0:1769077358.704000:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077358.704002:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077358.704011:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077358.704022:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961431680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077358.704031:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077358.704034:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077358.704035:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077358.704039:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3041 00000100:00080000:3.0:1769077358.704041:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077358.704044:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077358.704047:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077358.704053:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961431808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077358.704077:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961431552:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769077358.704154:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077358.704189:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077358/real 1769077358] req@ffff895e453d5500 x1855008961431552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077358.704204:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961431552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077358.704230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077358.704234:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077358.704237:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961431552:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077358.704243:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961431680:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077358.704262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077358.704270:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077358/real 1769077358] req@ffff895e453d7480 x1855008961431680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077358.704278:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961431680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077358.704284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077358.704287:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077358.704289:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961431680:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077358.704309:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961431808:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077358.704320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077358.704326:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077358/real 1769077358] req@ffff895e453d4e00 x1855008961431808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077358.704333:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961431808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077358.704338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077358.704340:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077358.704342:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961431808:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077359.215836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077359.407948:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077360.239853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077360.431920:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077361.263853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077361.456013:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077362.287830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077362.479899:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077363.311864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077363.503896:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077363.823931:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077363.823945:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077363.823951:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077363.823991:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3046 00000100:00080000:1.0:1769077363.824022:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077363.824031:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077363.824078:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961431936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077363.824121:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077363.824124:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077363.824125:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077363.824129:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3046 00000100:00080000:1.0:1769077363.824132:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077363.824136:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077363.824147:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077363.824188:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961432064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077363.824202:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077363.824208:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077363.824210:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077363.824213:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3046 00000100:00080000:1.0:1769077363.824217:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077363.824221:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077363.824225:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077363.824233:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961431936:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077363.824242:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961432192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077363.824321:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077363.824402:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077363/real 1769077363] req@ffff895e4713c380 x1855008961431936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077363.824418:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961431936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077363.824427:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077363.824431:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077363.824450:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961431936:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077363.824458:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961432064:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077363.824480:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077363.824491:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077363/real 1769077363] req@ffff895e4713df80 x1855008961432064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077363.824519:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961432064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077363.824528:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077363.824532:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077363.824535:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961432064:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077363.824590:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961432192:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077363.824605:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077363.824613:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077363/real 1769077363] req@ffff895e4713d180 x1855008961432192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077363.824633:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961432192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077363.824640:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077363.824643:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077363.824645:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961432192:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077364.335884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077364.527925:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077365.359813:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077365.551889:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077366.383851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077366.575909:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077367.407835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077367.599898:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077368.431870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077368.623944:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077368.943846:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077368.943857:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077368.943862:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077368.943872:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3051 00000100:00080000:1.0:1769077368.943883:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077368.943891:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077368.943930:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961432320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077368.943959:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077368.943964:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077368.943966:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077368.943971:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3051 00000100:00080000:1.0:1769077368.943976:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077368.943981:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077368.943991:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077368.944004:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961432320:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077368.944007:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961432448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077368.944024:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077368.944028:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077368.944030:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077368.944035:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3051 00000100:00080000:1.0:1769077368.944040:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077368.944045:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077368.944066:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077368.944078:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769077368.944084:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961432576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077368.944123:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077368/real 1769077368] req@ffff895e4713dc00 x1855008961432320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077368.944143:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961432320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077368.944154:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077368.944158:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077368.944162:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961432320:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077368.944187:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961432448:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077368.944208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077368.944220:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077368/real 1769077368] req@ffff895e4713f800 x1855008961432448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077368.944233:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961432448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077368.944254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077368.944258:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077368.944262:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961432448:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077368.944297:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961432576:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077368.944317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077368.944328:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077368/real 1769077368] req@ffff895e4713ca80 x1855008961432576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077368.944339:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961432576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077368.944348:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077368.944351:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077368.944355:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961432576:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077369.455851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077369.647912:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077370.479907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077370.671979:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077371.503836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077371.695866:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077372.527909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077372.719982:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077373.551836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077373.743863:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077374.063917:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077374.063928:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077374.063932:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077374.063940:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3056 00000100:00080000:2.0:1769077374.063950:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077374.063956:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077374.063989:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961432704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077374.064012:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077374.064015:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077374.064016:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077374.064019:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3056 00000100:00080000:2.0:1769077374.064023:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077374.064027:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077374.064035:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077374.064049:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961432832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077374.064075:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961432704:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077374.064081:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077374.064087:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077374.064089:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077374.064093:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3056 00000100:00080000:2.0:1769077374.064097:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077374.064100:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077374.064105:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077374.064116:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961432960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077374.064136:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077374.064170:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077374/real 1769077374] req@ffff895e44e29500 x1855008961432704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077374.064207:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961432704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077374.064225:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077374.064229:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077374.064232:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961432704:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077374.064251:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961432832:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077374.064269:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077374.064277:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077374/real 1769077374] req@ffff895e44e28a80 x1855008961432832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077374.064285:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961432832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077374.064291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077374.064293:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077374.064295:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961432832:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077374.064298:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961432960:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077374.064308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077374.064314:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077374/real 1769077374] req@ffff895e44e29180 x1855008961432960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077374.064321:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961432960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077374.064326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077374.064328:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077374.064329:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961432960:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077374.575864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077374.767904:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077375.599855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077375.791857:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077376.623886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077376.815910:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077377.647848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077377.839888:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077378.671852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077378.863939:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077379.183858:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077379.183870:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077379.183875:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077379.183885:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3061 00000100:00080000:2.0:1769077379.183894:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077379.183900:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077379.183938:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961433088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077379.183960:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077379.183962:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077379.183979:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077379.183984:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3061 00000100:00080000:2.0:1769077379.183989:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077379.183994:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077379.184004:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077379.184009:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961433088:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077379.184032:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961433216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077379.184048:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077379.184051:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077379.184054:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077379.184058:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3061 00000100:00080000:2.0:1769077379.184063:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077379.184067:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:2.0:1769077379.184072:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077379.184079:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077379.184109:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961433344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077379.184131:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077379/real 1769077379] req@ffff895e44e2bb80 x1855008961433088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077379.184147:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961433088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077379.184156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077379.184160:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077379.184163:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961433088:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077379.184185:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961433216:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077379.184202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077379.184210:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077379/real 1769077379] req@ffff895e44e2b800 x1855008961433216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077379.184218:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961433216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077379.184224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077379.184226:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077379.184229:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961433216:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077379.184232:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961433344:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077379.184254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077379.184260:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077379/real 1769077379] req@ffff895e44e29c00 x1855008961433344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077379.184268:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961433344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077379.184273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077379.184276:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077379.184278:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961433344:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077379.695847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077379.887892:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077380.719848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077380.911929:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077381.743809:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077381.935918:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077382.767853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077382.959905:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077383.791924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077383.983899:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077384.303870:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077384.303878:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077384.303882:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077384.303890:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3066 00000100:00080000:1.0:1769077384.303900:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077384.303905:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077384.303936:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961433472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077384.303957:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077384.303960:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077384.303961:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077384.303963:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3066 00000100:00080000:1.0:1769077384.303966:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077384.303981:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077384.303989:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077384.303999:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961433600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077384.304019:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077384.304023:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077384.304026:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077384.304030:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961433472:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077384.304030:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3066 00000100:00080000:1.0:1769077384.304036:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077384.304043:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077384.304048:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077384.304063:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961433728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077384.304105:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077384.304137:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077384/real 1769077384] req@ffff895e4713f480 x1855008961433472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077384.304152:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961433472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077384.304160:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077384.304163:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077384.304165:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961433472:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077384.304184:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961433600:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077384.304200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077384.304208:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077384/real 1769077384] req@ffff895e4713ce00 x1855008961433600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077384.304228:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961433600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077384.304234:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077384.304237:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077384.304239:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961433600:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077384.304243:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961433728:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077384.304257:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077384.304263:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077384/real 1769077384] req@ffff895e4713d500 x1855008961433728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077384.304270:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961433728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077384.304275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077384.304277:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077384.304279:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961433728:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077384.815851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077385.007896:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077385.839843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077386.031937:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077386.863898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077387.055915:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077387.887864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077388.079901:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077388.911848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077389.103881:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077389.423821:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077389.423830:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077389.423833:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077389.423849:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3071 00000100:00080000:1.0:1769077389.423859:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077389.423865:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077389.423896:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961433856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077389.423915:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077389.423941:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077389.423943:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077389.423947:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3071 00000100:00080000:1.0:1769077389.423950:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077389.423954:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077389.423961:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077389.423971:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961433856:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077389.423973:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961433984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077389.423987:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077389.423990:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077389.423992:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077389.423996:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3071 00000100:00080000:1.0:1769077389.424000:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077389.424004:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077389.424009:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077389.424022:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961434112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077389.424044:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077389.424084:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077389/real 1769077389] req@ffff895e4713ea00 x1855008961433856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077389.424106:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961433856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077389.424114:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077389.424117:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077389.424120:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961433856:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077389.424139:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961433984:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077389.424156:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077389.424165:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077389/real 1769077389] req@ffff895e4713f100 x1855008961433984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077389.424173:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961433984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077389.424178:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077389.424180:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077389.424183:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961433984:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077389.424186:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961434112:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077389.424197:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077389.424202:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077389/real 1769077389] req@ffff895e4713e300 x1855008961434112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077389.424209:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961434112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077389.424223:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077389.424226:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077389.424228:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961434112:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077389.935901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077390.127961:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077390.959874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077391.151856:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077391.983848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077392.175952:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077393.007848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077393.199891:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077394.031897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077394.223984:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077394.543911:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077394.543923:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077394.543948:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077394.543959:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3077 00000100:00080000:1.0:1769077394.543981:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077394.543988:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077394.544031:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961434240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077394.544061:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077394.544063:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077394.544068:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077394.544072:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3077 00000100:00080000:1.0:1769077394.544074:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077394.544078:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077394.544087:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077394.544096:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961434368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077394.544105:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077394.544108:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077394.544109:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077394.544111:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3077 00000100:00080000:1.0:1769077394.544113:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077394.544117:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077394.544122:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077394.544123:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961434240:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077394.544131:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961434496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077394.544211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077394.544253:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077394/real 1769077394] req@ffff895e4713d880 x1855008961434240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077394.544270:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961434240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077394.544278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077394.544282:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077394.544285:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961434240:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077394.544291:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961434368:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077394.544309:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077394.544317:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077394/real 1769077394] req@ffff895e4713ed80 x1855008961434368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077394.544325:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961434368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077394.544341:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077394.544344:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077394.544346:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961434368:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077394.544378:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961434496:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077394.544402:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077394.544414:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077394/real 1769077394] req@ffff895e4713c000 x1855008961434496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077394.544426:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961434496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077394.544434:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077394.544437:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077394.544441:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961434496:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077395.056001:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077395.247923:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077396.079881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077396.271981:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077397.103854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077397.296008:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077398.127882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077398.319925:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077399.151847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077399.343942:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077399.663896:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077399.663908:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077399.663913:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077399.663923:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3082 00000100:00080000:3.0:1769077399.663934:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077399.663941:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077399.663980:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961434624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077399.664010:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077399.664021:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077399.664023:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077399.664028:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3082 00000100:00080000:3.0:1769077399.664033:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077399.664043:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077399.664054:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077399.664055:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961434624:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077399.664075:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961434752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077399.664088:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077399.664092:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077399.664108:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077399.664114:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3082 00000100:00080000:3.0:1769077399.664119:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077399.664124:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077399.664131:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077399.664148:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961434880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077399.664157:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961434752:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077399.664200:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961434880:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077400.175882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077400.367912:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077401.199820:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077401.391910:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077402.223849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077402.415895:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769077402.736005:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769077402.736062:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077402.736080:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077402.736126:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077402.736137:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769077402.736232:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077399/real 1769077402] req@ffff895e453d4a80 x1855008961434880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077402.736273:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961434880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077402.736285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077402.736291:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077402.736295:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961434880:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077402.736305:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077399/real 1769077402] req@ffff895e453d5c00 x1855008961434752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077402.736317:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961434752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077402.736326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077402.736330:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077402.736332:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961434752:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077402.736340:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077399/real 1769077402] req@ffff895e453d7800 x1855008961434624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077402.736352:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961434624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077402.736359:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077402.736362:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077402.736366:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961434624:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077403.247912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077403.439950:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077404.271871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077404.463968:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077404.783960:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077404.784020:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077404.784025:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077404.784038:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3087 00000100:00080000:0.0:1769077404.784053:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077404.784063:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077404.784111:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961435008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077404.784176:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077404.784186:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077404.784189:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077404.784194:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3087 00000100:00080000:0.0:1769077404.784199:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077404.784204:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077404.784216:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077404.784232:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961435136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077404.784243:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077404.784248:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077404.784250:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077404.784254:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3087 00000100:00080000:0.0:1769077404.784258:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077404.784263:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077404.784268:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077404.784283:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961435264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077404.784416:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961435008:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769077404.784532:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077404.784587:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077404/real 1769077404] req@ffff895e4383f800 x1855008961435008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077404.784610:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961435008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077404.784624:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077404.784630:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077404.784636:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961435008:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077404.784645:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961435136:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077404.784705:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077404.784722:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077404/real 1769077404] req@ffff895e4383e300 x1855008961435136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077404.784857:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961435136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077404.784869:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077404.784873:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077404.784878:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961435136:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077404.784886:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961435264:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077404.784914:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077404.784928:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077404/real 1769077404] req@ffff895e4383d500 x1855008961435264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077404.784943:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961435264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077404.784952:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077404.784956:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077404.784961:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961435264:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077405.295854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077405.487918:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077406.319857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077406.511906:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077407.343918:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077407.535926:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077408.367894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077408.559986:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077409.391872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077409.583992:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077409.903834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077409.903846:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077409.903851:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077409.903862:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3092 00000100:00080000:0.0:1769077409.903875:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077409.903883:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077409.903986:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961435392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077409.904007:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077409.904010:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077409.904011:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077409.904014:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3092 00000100:00080000:0.0:1769077409.904017:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077409.904019:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077409.904027:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077409.904036:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961435520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077409.904044:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077409.904047:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077409.904048:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077409.904053:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3092 00000100:00100000:3.0:1769077409.904055:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961435392:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077409.904056:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077409.904063:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077409.904081:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077409.904092:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961435648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077409.904118:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077409.904156:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077409/real 1769077409] req@ffff895e4383d500 x1855008961435392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077409.904190:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961435392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077409.904198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077409.904201:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077409.904204:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961435392:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077409.904223:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961435520:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077409.904239:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077409.904247:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077409/real 1769077409] req@ffff895e4383e300 x1855008961435520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077409.904255:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961435520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077409.904261:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077409.904263:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077409.904265:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961435520:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077409.904268:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961435648:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077409.904278:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077409.904283:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077409/real 1769077409] req@ffff895e4383f800 x1855008961435648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077409.904291:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961435648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077409.904304:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077409.904307:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077409.904309:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961435648:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077410.415836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077410.607875:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077411.439908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077411.631944:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077412.463893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077412.655887:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077413.487927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077413.680036:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077414.511849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077414.703896:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077415.023862:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077415.023874:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077415.023879:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077415.023889:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3097 00000100:00080000:0.0:1769077415.023902:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077415.023909:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077415.023950:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961435776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077415.023979:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077415.023981:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077415.023983:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077415.023986:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3097 00000100:00080000:0.0:1769077415.023988:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077415.023991:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077415.024000:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077415.024009:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961435904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077415.024017:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077415.024020:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077415.024038:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077415.024042:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3097 00000100:00080000:0.0:1769077415.024045:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077415.024049:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077415.024054:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077415.024060:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961435776:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077415.024064:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961436032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077415.024146:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077415.024202:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077415/real 1769077415] req@ffff895e4383ca80 x1855008961435776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077415.024225:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961435776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077415.024236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077415.024241:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077415.024245:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961435776:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077415.024254:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961435904:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077415.024285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077415.024299:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077415/real 1769077415] req@ffff895e4383dc00 x1855008961435904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077415.024313:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961435904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077415.024342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077415.024346:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077415.024350:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961435904:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077415.024376:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961436032:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077415.024399:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077415.024411:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077415/real 1769077415] req@ffff895e4383ea00 x1855008961436032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077415.024434:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961436032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077415.024442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077415.024446:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077415.024449:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961436032:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077415.535884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077415.728014:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077416.559840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077416.751913:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077417.583836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077417.775896:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077418.607826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077418.799908:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077419.631844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077419.823903:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077420.143876:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077420.143884:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077420.143888:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077420.143896:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3102 00000100:00080000:2.0:1769077420.143905:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077420.143912:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077420.143945:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961436160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077420.143968:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077420.143971:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077420.143972:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077420.143974:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3102 00000100:00080000:2.0:1769077420.143977:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077420.143979:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077420.143986:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077420.143995:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961436288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077420.144010:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961436160:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077420.144013:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077420.144019:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077420.144033:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077420.144037:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3102 00000100:00080000:2.0:1769077420.144041:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077420.144045:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077420.144049:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077420.144059:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961436416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077420.144082:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077420.144114:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077420/real 1769077420] req@ffff895e44e29c00 x1855008961436160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077420.144136:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961436160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077420.144154:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077420.144157:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077420.144160:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961436160:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077420.144180:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961436288:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077420.144197:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077420.144206:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077420/real 1769077420] req@ffff895e44e2b800 x1855008961436288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077420.144214:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961436288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077420.144220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077420.144222:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077420.144224:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961436288:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077420.144228:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961436416:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077420.144238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077420.144258:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077420/real 1769077420] req@ffff895e44e2bb80 x1855008961436416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077420.144269:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961436416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077420.144277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077420.144280:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077420.144284:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961436416:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077420.655832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077420.847888:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077421.679849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077421.871893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077422.703850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077422.895904:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077423.727828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077423.919871:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077424.751845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077424.943910:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077425.263812:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077425.263822:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077425.263825:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077425.263834:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3107 00000100:00080000:2.0:1769077425.263843:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077425.263848:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077425.263878:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961436544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077425.263900:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077425.263902:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077425.263903:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077425.263906:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3107 00000100:00080000:2.0:1769077425.263908:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077425.263911:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077425.263918:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077425.263926:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961436672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077425.263946:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961436544:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077425.263949:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077425.263955:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077425.263956:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077425.263959:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3107 00000100:00080000:2.0:1769077425.263962:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077425.263966:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077425.263970:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077425.263980:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961436800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077425.264005:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077425.264041:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077425/real 1769077425] req@ffff895e44e29180 x1855008961436544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077425.264061:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961436544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077425.264073:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077425.264077:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077425.264082:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961436544:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077425.264112:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961436672:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077425.264132:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077425.264141:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077425/real 1769077425] req@ffff895e44e28a80 x1855008961436672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077425.264152:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961436672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077425.264161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077425.264164:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077425.264168:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961436672:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077425.264183:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961436800:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077425.264206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077425.264216:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077425/real 1769077425] req@ffff895e44e29500 x1855008961436800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077425.264225:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961436800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077425.264245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077425.264249:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077425.264253:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961436800:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077425.775848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077425.967927:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077426.799887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077426.991954:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077427.823837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077428.015887:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077428.847845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077429.039908:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077429.871862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077430.063891:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077430.383855:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077430.383880:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077430.383884:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077430.383892:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3112 00000100:00080000:1.0:1769077430.383901:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077430.383906:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077430.383933:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961436928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077430.383960:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077430.383964:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077430.383967:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077430.383971:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3112 00000100:00080000:1.0:1769077430.383975:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077430.383980:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077430.383990:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077430.384002:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961436928:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077430.384006:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961437056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077430.384026:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077430.384046:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077430.384047:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077430.384051:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3112 00000100:00080000:1.0:1769077430.384055:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077430.384058:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077430.384063:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077430.384074:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961437184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077430.384118:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077430.384179:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077430/real 1769077430] req@ffff895e4713fb80 x1855008961436928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077430.384199:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961436928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077430.384210:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077430.384215:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077430.384218:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961436928:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077430.384308:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961437056:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077430.384327:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077430.384337:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077430/real 1769077430] req@ffff895e4713c700 x1855008961437056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077430.384347:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961437056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077430.384356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077430.384359:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077430.384363:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961437056:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077430.384369:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961437184:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077430.384389:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077430.384400:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077430/real 1769077430] req@ffff895e4713c000 x1855008961437184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077430.384412:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961437184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077430.384418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077430.384420:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077430.384422:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961437184:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077430.895821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077431.087927:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077431.919835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077432.111868:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077432.943854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077433.135915:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077433.967844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077434.159886:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077434.991864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077435.183919:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077435.503806:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077435.503814:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077435.503824:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077435.503832:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3118 00000100:00080000:1.0:1769077435.503842:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077435.503847:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077435.503880:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961437312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077435.503900:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077435.503903:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077435.503904:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077435.503907:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3118 00000100:00080000:1.0:1769077435.503910:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077435.503912:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077435.503919:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077435.503928:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961437440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077435.503936:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077435.503939:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077435.503942:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077435.503944:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961437312:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077435.503946:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3118 00000100:00080000:1.0:1769077435.503950:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077435.503967:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077435.503971:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077435.503983:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961437568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077435.504006:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077435.504038:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077435/real 1769077435] req@ffff895e4713ed80 x1855008961437312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077435.504052:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961437312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077435.504059:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077435.504063:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077435.504065:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961437312:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077435.504084:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961437440:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077435.504098:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077435.504106:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077435/real 1769077435] req@ffff895e4713d880 x1855008961437440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077435.504113:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961437440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077435.504120:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077435.504122:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077435.504124:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961437440:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077435.504129:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961437568:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077435.504174:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077435.504187:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077435/real 1769077435] req@ffff895e4713e300 x1855008961437568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077435.504200:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961437568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077435.504208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077435.504211:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077435.504213:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961437568:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077436.015841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077436.207918:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077437.039884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077437.231858:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077438.063826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077438.255905:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077439.087868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077439.279849:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077440.111867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077440.303944:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077440.623847:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077440.623858:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077440.623863:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077440.623873:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3123 00000100:00080000:1.0:1769077440.623883:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077440.623890:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077440.623925:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961437696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077440.623953:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077440.623955:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077440.623956:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077440.623959:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3123 00000100:00080000:1.0:1769077440.623962:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077440.623965:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077440.623972:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077440.623980:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961437824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077440.623987:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077440.623990:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077440.624005:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077440.624009:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3123 00000100:00080000:1.0:1769077440.624012:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077440.624015:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077440.624019:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077440.624030:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961437696:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077440.624031:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961437952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077440.624156:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077440.624204:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077440/real 1769077440] req@ffff895e4713f100 x1855008961437696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077440.624222:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961437696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077440.624231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077440.624236:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077440.624239:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961437696:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077440.624258:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961437824:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077440.624277:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077440.624286:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077440/real 1769077440] req@ffff895e4713ea00 x1855008961437824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077440.624295:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961437824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077440.624301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077440.624303:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077440.624306:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961437824:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077440.624332:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961437952:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077440.624343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077440.624349:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077440/real 1769077440] req@ffff895e4713d500 x1855008961437952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077440.624356:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961437952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077440.624362:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077440.624364:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077440.624366:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961437952:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077441.135850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077441.327869:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077442.159858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077442.351916:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077443.183861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077443.375849:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077444.207872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077444.399947:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077445.231852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077445.423826:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077445.743843:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077445.743855:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077445.743859:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077445.743869:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3128 00000100:00080000:1.0:1769077445.743880:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077445.743886:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077445.743917:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961438080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077445.743939:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077445.743942:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077445.743943:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077445.743946:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3128 00000100:00080000:1.0:1769077445.743948:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077445.743950:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077445.743957:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077445.743966:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961438208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077445.744016:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961438080:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077445.744022:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077445.744026:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077445.744027:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077445.744031:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3128 00000100:00080000:1.0:1769077445.744034:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077445.744038:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077445.744043:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077445.744054:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961438336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077445.744089:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077445.744121:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077445/real 1769077445] req@ffff895e4713ce00 x1855008961438080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077445.744136:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961438080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077445.744144:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077445.744147:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077445.744150:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961438080:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077445.744169:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961438208:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077445.744184:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077445.744192:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077445/real 1769077445] req@ffff895e4713f480 x1855008961438208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077445.744199:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961438208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077445.744205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077445.744207:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077445.744209:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961438208:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077445.744213:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961438336:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077445.744222:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077445.744237:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077445/real 1769077445] req@ffff895e4713ca80 x1855008961438336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077445.744245:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961438336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077445.744250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077445.744252:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077445.744255:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961438336:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077446.255889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077446.447916:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077447.279866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077447.471896:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077448.303849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077448.495908:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077449.327840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077449.519904:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077450.351928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077450.543875:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077450.863829:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077450.863837:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077450.863840:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077450.863848:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3133 00000100:00080000:1.0:1769077450.863858:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077450.863864:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077450.863922:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961438464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077450.863950:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077450.863952:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077450.863954:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077450.863957:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3133 00000100:00080000:1.0:1769077450.863959:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077450.863977:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077450.863986:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077450.863991:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961438464:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077450.864000:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961438592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077450.864009:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077450.864012:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077450.864014:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077450.864016:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3133 00000100:00080000:1.0:1769077450.864019:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077450.864022:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077450.864025:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077450.864033:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961438720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077450.864057:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077450.864113:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077450/real 1769077450] req@ffff895e4713f800 x1855008961438464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077450.864129:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961438464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077450.864137:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077450.864140:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077450.864144:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961438464:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077450.864164:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961438592:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077450.864193:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077450.864211:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077450/real 1769077450] req@ffff895e4713dc00 x1855008961438592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077450.864225:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961438592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077450.864236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077450.864240:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077450.864243:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961438592:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077450.864250:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961438720:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077450.864266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077450.864274:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077450/real 1769077450] req@ffff895e4713d180 x1855008961438720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077450.864282:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961438720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077450.864287:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077450.864289:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077450.864292:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961438720:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077451.375854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077451.567920:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077452.399821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077452.591905:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077453.423822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077453.615951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077454.447874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077454.639924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077455.471853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077455.663909:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077455.983875:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077455.983885:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077455.983889:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077455.983896:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3138 00000100:00080000:0.0:1769077455.983907:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077455.983914:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077455.983952:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961438848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077455.983979:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077455.983981:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077455.983983:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077455.983986:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3138 00000100:00080000:0.0:1769077455.983989:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077455.983992:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077455.984013:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077455.984017:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961438848:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077455.984027:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961438976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077455.984037:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077455.984039:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077455.984041:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077455.984044:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3138 00000100:00080000:0.0:1769077455.984048:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077455.984051:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077455.984054:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077455.984062:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961439104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077455.984076:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077455.984110:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077455/real 1769077455] req@ffff895e4383c000 x1855008961438848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077455.984124:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961438848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077455.984133:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077455.984137:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077455.984142:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961438848:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077455.984178:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961438976:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077455.984204:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077455.984217:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077455/real 1769077455] req@ffff895e4383f480 x1855008961438976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077455.984231:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961438976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077455.984238:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077455.984240:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077455.984243:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961438976:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077455.984248:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961439104:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077455.984263:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077455.984270:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077455/real 1769077455] req@ffff895e4383ce00 x1855008961439104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077455.984299:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961439104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077455.984305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077455.984307:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077455.984309:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961439104:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077456.495881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077456.687950:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077457.519839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077457.711925:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077458.543817:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077458.735869:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077459.567844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077459.759921:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077460.591852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077460.783908:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077461.103869:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077461.103878:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077461.103882:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077461.103891:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3143 00000100:00080000:3.0:1769077461.103900:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077461.103906:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077461.103938:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961439232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077461.103962:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077461.103965:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077461.103966:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077461.103968:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3143 00000100:00080000:3.0:1769077461.103970:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077461.103973:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077461.103980:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077461.103991:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961439360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077461.104001:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961439232:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077461.104016:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077461.104018:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077461.104020:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077461.104024:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3143 00000100:00080000:3.0:1769077461.104027:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077461.104029:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077461.104033:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077461.104043:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961439488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077461.104072:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077461.104115:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077461/real 1769077461] req@ffff895e453d5f80 x1855008961439232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077461.104135:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961439232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077461.104143:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077461.104146:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077461.104149:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961439232:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077461.104172:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961439360:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077461.104205:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077461.104219:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077461/real 1769077461] req@ffff895e453d7800 x1855008961439360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077461.104231:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961439360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077461.104237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077461.104240:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077461.104242:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961439360:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077461.104246:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961439488:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077461.104261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077461.104268:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077461/real 1769077461] req@ffff895e453d5c00 x1855008961439488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077461.104276:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961439488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077461.104281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077461.104283:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077461.104285:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961439488:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077461.615923:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077461.807868:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077462.639849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077462.831877:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077463.663851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077463.855883:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077464.687855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077464.879834:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077465.711849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077465.903898:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077466.223823:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077466.223832:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077466.223836:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077466.223844:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3148 00000100:00080000:1.0:1769077466.223853:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077466.223859:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077466.223890:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961439616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077466.223922:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077466.223924:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077466.223926:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077466.223928:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3148 00000100:00080000:1.0:1769077466.223931:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077466.223934:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077466.223941:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077466.223950:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961439744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077466.223957:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077466.223962:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769077466.223963:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961439616:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077466.223964:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077466.223972:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3148 00000100:00080000:1.0:1769077466.223974:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077466.223977:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077466.223982:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077466.223997:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961439872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077466.224090:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961439744:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077466.224135:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961439872:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077466.735878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077466.927880:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077467.759849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077467.951878:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077468.783846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077468.975877:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1769077469.296060:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:0.0:1769077469.296108:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1769077469.296127:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:0.0:1769077469.296173:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:0.0:1769077469.296182:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769077469.296269:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077466/real 1769077469] req@ffff895e4713c380 x1855008961439744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077469.296296:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961439744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077469.296310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077469.296316:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077469.296338:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961439744:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077469.296348:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077466/real 1769077469] req@ffff895e4713d180 x1855008961439872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077469.296363:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961439872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077469.296387:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077469.296392:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077469.296396:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961439872:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077469.296403:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077466/real 1769077469] req@ffff895e4713df80 x1855008961439616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077469.296419:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961439616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077469.296427:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077469.296431:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077469.296434:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961439616:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077469.807851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077469.999918:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077470.831854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077471.023843:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077471.343897:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077471.343908:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077471.343912:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077471.343920:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3153 00000100:00080000:3.0:1769077471.343930:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077471.343937:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077471.343993:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961440000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077471.344031:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077471.344035:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077471.344037:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077471.344058:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3153 00000100:00080000:3.0:1769077471.344064:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077471.344068:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077471.344078:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077471.344092:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961440128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077471.344103:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077471.344106:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077471.344108:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077471.344112:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3153 00000100:00080000:3.0:1769077471.344115:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077471.344118:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077471.344122:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077471.344132:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961440256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077471.344236:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961440000:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769077471.344322:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077471.344362:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077471/real 1769077471] req@ffff895e453d4a80 x1855008961440000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077471.344381:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961440000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077471.344392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077471.344397:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077471.344401:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961440000:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077471.344408:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961440128:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077471.344433:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077471.344447:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077471/real 1769077471] req@ffff895e453d4e00 x1855008961440128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077471.344460:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961440128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077471.344482:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077471.344486:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077471.344489:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961440128:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077471.344496:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961440256:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077471.344530:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077471.344557:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077471/real 1769077471] req@ffff895e453d7480 x1855008961440256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077471.344571:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961440256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077471.344580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077471.344584:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077471.344587:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961440256:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077471.855847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077472.047900:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077472.879830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077473.071861:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077473.903864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077474.095893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077474.927857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077475.119928:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077475.951847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077476.143909:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077476.463875:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077476.463886:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077476.463891:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077476.463901:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3159 00000100:00080000:3.0:1769077476.463913:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077476.463920:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077476.463957:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961440384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077476.463999:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077476.464002:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077476.464004:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077476.464007:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3159 00000100:00080000:3.0:1769077476.464011:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077476.464014:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077476.464021:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077476.464034:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961440512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077476.464041:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077476.464044:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077476.464045:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077476.464048:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3159 00000100:00080000:3.0:1769077476.464050:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077476.464053:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077476.464055:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077476.464064:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961440384:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077476.464065:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961440640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077476.464146:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077476.464186:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077476/real 1769077476] req@ffff895e453d5500 x1855008961440384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077476.464206:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961440384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077476.464233:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077476.464238:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077476.464243:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961440384:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077476.464251:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961440512:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077476.464278:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077476.464291:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077476/real 1769077476] req@ffff895e453d7b80 x1855008961440512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077476.464306:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961440512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077476.464315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077476.464318:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077476.464322:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961440512:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077476.464358:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961440640:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077476.464381:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077476.464393:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077476/real 1769077476] req@ffff895e453d7100 x1855008961440640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077476.464407:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961440640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077476.464414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077476.464417:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077476.464420:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961440640:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077476.975859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077477.167871:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077477.999853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077478.191920:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077479.023823:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077479.215919:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077480.047868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077480.239868:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077481.071865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077481.263928:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077481.583820:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077481.583828:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077481.583832:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077481.583854:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3164 00000100:00080000:3.0:1769077481.583863:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077481.583869:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077481.583896:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961440768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077481.583921:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077481.583924:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077481.583926:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077481.583930:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3164 00000100:00080000:3.0:1769077481.583934:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077481.583938:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077481.583947:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077481.583967:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961440896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077481.583971:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961440768:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077481.583978:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077481.583981:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077481.583983:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077481.583987:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3164 00000100:00080000:3.0:1769077481.583991:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077481.583994:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077481.583999:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077481.584007:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961441024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077481.584032:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077481.584067:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077481/real 1769077481] req@ffff895e453d4700 x1855008961440768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077481.584082:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961440768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077481.584089:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077481.584092:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077481.584095:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961440768:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077481.584114:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961440896:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077481.584131:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077481.584139:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077481/real 1769077481] req@ffff895e453d5880 x1855008961440896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077481.584147:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961440896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077481.584153:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077481.584155:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077481.584157:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961440896:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077481.584167:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961441024:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077481.584180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077481.584187:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077481/real 1769077481] req@ffff895e453d5180 x1855008961441024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077481.584195:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961441024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077481.584200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077481.584202:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077481.584204:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961441024:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077482.095867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077482.287899:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077483.119879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077483.311910:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077484.143866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077484.335877:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077485.167881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077485.359925:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077486.191864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077486.383883:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077486.703911:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077486.703926:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077486.703932:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077486.703945:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3169 00000100:00080000:2.0:1769077486.703959:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077486.703967:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077486.704016:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961441152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077486.704052:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077486.704057:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077486.704059:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077486.704063:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3169 00000100:00080000:2.0:1769077486.704067:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077486.704090:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077486.704102:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077486.704115:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961441280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077486.704126:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077486.704130:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077486.704133:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077486.704137:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3169 00000100:00080000:2.0:1769077486.704142:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077486.704149:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077486.704151:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961441152:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769077486.704154:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077486.704166:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961441408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077486.704248:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077486.704330:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077486/real 1769077486] req@ffff895e44e28e00 x1855008961441152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077486.704350:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961441152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077486.704359:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077486.704362:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077486.704366:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961441152:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077486.704380:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961441280:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077486.704400:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077486.704410:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077486/real 1769077486] req@ffff895e44e2b480 x1855008961441280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077486.704418:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961441280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077486.704424:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077486.704427:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077486.704429:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961441280:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077486.704456:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961441408:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077486.704467:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077486.704473:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077486/real 1769077486] req@ffff895e44e29f80 x1855008961441408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077486.704480:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961441408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077486.704486:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077486.704488:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077486.704490:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961441408:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077487.215965:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077487.407993:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077488.239860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077488.431924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077489.263846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077489.455878:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077490.287854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077490.479880:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077491.311845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077491.503897:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077491.823880:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077491.823891:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077491.823896:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077491.823907:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3174 00000100:00080000:1.0:1769077491.823920:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077491.823927:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077491.823967:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961441536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077491.824000:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077491.824004:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077491.824006:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077491.824011:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3174 00000100:00080000:1.0:1769077491.824015:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077491.824019:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077491.824029:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077491.824042:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961441664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077491.824052:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961441536:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077491.824079:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077491.824084:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077491.824086:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077491.824092:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3174 00000100:00080000:1.0:1769077491.824098:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077491.824103:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:0.0:1769077491.824109:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:1.0:1769077491.824110:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077491.824143:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077491/real 1769077491] req@ffff895e4713dc00 x1855008961441536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077491.824163:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961441536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077491.824171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077491.824174:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077491.824180:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961441536:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077491.824192:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961441792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077491.824223:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961441664:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077491.824252:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077491.824264:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077491/real 1769077491] req@ffff895e4713f800 x1855008961441664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077491.824273:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961441664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077491.824280:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077491.824282:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077491.824284:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961441664:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077491.824294:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961441792:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077491.824306:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077491.824312:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077491/real 1769077491] req@ffff895e4713ca80 x1855008961441792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077491.824320:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961441792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077491.824334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077491.824337:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077491.824339:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961441792:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077492.335957:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077492.527829:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077493.359820:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077493.551885:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077494.383818:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077494.575879:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077495.407858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077495.599857:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077496.431845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077496.623988:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077496.943934:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077496.943950:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077496.943955:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077496.943968:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3179 00000100:00080000:2.0:1769077496.943983:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077496.944003:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077496.944073:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961441920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077496.944152:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077496.944157:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077496.944160:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077496.944166:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3179 00000100:00080000:2.0:1769077496.944171:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077496.944176:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077496.944188:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961441920:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769077496.944190:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077496.944212:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961442048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077496.944225:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077496.944229:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077496.944232:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077496.944236:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3179 00000100:00080000:2.0:1769077496.944252:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077496.944258:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077496.944264:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077496.944274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769077496.944280:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961442176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077496.944326:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077496/real 1769077496] req@ffff895e44e2a300 x1855008961441920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077496.944347:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961441920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077496.944364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077496.944370:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077496.944374:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961441920:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077496.944403:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961442048:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077496.944425:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077496.944436:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077496/real 1769077496] req@ffff895e44e2aa00 x1855008961442048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077496.944449:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961442048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077496.944459:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077496.944463:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077496.944466:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961442048:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077496.944480:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961442176:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077496.944499:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077496.944508:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077496/real 1769077496] req@ffff895e44e2b100 x1855008961442176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077496.944521:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961442176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077496.944530:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077496.944533:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077496.944537:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961442176:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077497.455917:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077497.647956:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077498.479892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077498.671918:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077499.503896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077499.695970:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077500.527897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077500.719929:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077501.551884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077501.743942:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077502.063927:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077502.063944:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077502.063950:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077502.063961:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3184 00000100:00080000:2.0:1769077502.063974:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077502.063981:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077502.064057:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961442304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077502.064094:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077502.064099:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077502.064101:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077502.064106:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3184 00000100:00080000:2.0:1769077502.064110:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077502.064115:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077502.064127:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077502.064142:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961442432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077502.064154:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077502.064159:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077502.064161:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077502.064166:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961442304:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077502.064167:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3184 00000100:00080000:2.0:1769077502.064174:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077502.064180:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077502.064207:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077502.064222:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961442560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077502.064260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077502.064312:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077502/real 1769077502] req@ffff895e44e28000 x1855008961442304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077502.064334:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961442304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077502.064348:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077502.064353:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077502.064358:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961442304:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077502.064390:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961442432:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077502.064417:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077502.064432:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077502/real 1769077502] req@ffff895e44e28380 x1855008961442432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077502.064456:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961442432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077502.064466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077502.064470:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077502.064474:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961442432:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077502.064481:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961442560:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077502.064518:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077502.064532:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077502/real 1769077502] req@ffff895e44e29880 x1855008961442560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077502.064545:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961442560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077502.064555:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077502.064559:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077502.064563:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961442560:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077502.575883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077502.767953:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077503.599862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077503.791938:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077504.623873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077504.815867:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077505.647917:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077505.839899:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077506.671872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077506.863936:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077507.183860:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077507.183871:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077507.183877:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077507.183890:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3189 00000100:00080000:0.0:1769077507.183905:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077507.183913:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077507.183990:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961442688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077507.184061:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077507.184066:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077507.184068:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077507.184073:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3189 00000100:00080000:0.0:1769077507.184078:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077507.184084:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077507.184096:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077507.184111:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961442688:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077507.184116:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961442816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077507.184127:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077507.184130:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077507.184132:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077507.184135:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3189 00000100:00080000:0.0:1769077507.184139:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077507.184142:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077507.184146:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077507.184157:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961442944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077507.184204:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077507.184258:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077507/real 1769077507] req@ffff895e4383c380 x1855008961442688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077507.184279:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961442688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077507.184288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077507.184292:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077507.184295:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961442688:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077507.184320:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961442816:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077507.184347:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077507.184356:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077507/real 1769077507] req@ffff895e4383f100 x1855008961442816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077507.184365:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961442816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077507.184383:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077507.184386:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077507.184388:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961442816:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077507.184393:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961442944:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077507.184407:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077507.184414:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077507/real 1769077507] req@ffff895e4383fb80 x1855008961442944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077507.184422:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961442944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077507.184428:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077507.184430:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077507.184432:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961442944:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077507.695860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077507.887843:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077508.719862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077508.911946:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077509.743874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077509.935917:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077510.767911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077510.959837:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077511.791827:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077511.983882:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077512.303824:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077512.303832:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077512.303836:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077512.303843:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3194 00000100:00080000:0.0:1769077512.303892:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077512.303898:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077512.303927:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961443072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077512.303952:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077512.303954:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077512.303956:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077512.303958:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3194 00000100:00080000:0.0:1769077512.303961:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077512.303963:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077512.303975:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077512.303985:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961443200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077512.303993:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077512.303994:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961443072:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077512.303999:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077512.304003:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077512.304007:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3194 00000100:00080000:0.0:1769077512.304008:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077512.304011:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077512.304014:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077512.304023:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961443328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077512.304055:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077512.304086:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077512/real 1769077512] req@ffff895e4383c700 x1855008961443072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077512.304099:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961443072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077512.304107:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077512.304117:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077512.304120:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961443072:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077512.304139:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961443200:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077512.304155:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077512.304163:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077512/real 1769077512] req@ffff895e4383e680 x1855008961443200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077512.304171:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961443200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077512.304177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077512.304179:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077512.304182:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961443200:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077512.304185:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961443328:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077512.304195:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077512.304201:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077512/real 1769077512] req@ffff895e4383d880 x1855008961443328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077512.304214:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961443328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077512.304219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077512.304221:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077512.304223:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961443328:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077512.815952:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077513.007979:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077513.839844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077514.031927:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077514.863862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077515.055893:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077515.887833:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077516.079930:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077516.911861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077517.103893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077517.423932:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077517.423941:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077517.423945:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077517.423952:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3199 00000100:00080000:3.0:1769077517.423962:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077517.423968:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077517.424003:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961443456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077517.424033:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077517.424037:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077517.424039:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077517.424044:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3199 00000100:00080000:3.0:1769077517.424048:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077517.424053:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077517.424068:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077517.424069:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961443456:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077517.424082:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961443584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077517.424092:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077517.424094:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077517.424096:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077517.424099:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3199 00000100:00080000:3.0:1769077517.424102:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077517.424105:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077517.424122:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077517.424128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077517.424133:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961443712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077517.424162:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077517/real 1769077517] req@ffff895e453d6680 x1855008961443456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077517.424178:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961443456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077517.424186:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077517.424189:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077517.424192:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961443456:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077517.424210:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961443584:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077517.424224:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077517.424232:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077517/real 1769077517] req@ffff895e453d4380 x1855008961443584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077517.424239:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961443584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077517.424245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077517.424247:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077517.424249:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961443584:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077517.424252:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961443712:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077517.424272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077517.424279:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077517/real 1769077517] req@ffff895e453d6d80 x1855008961443712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077517.424287:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961443712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077517.424292:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077517.424293:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077517.424295:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961443712:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077517.935896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077518.128020:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077518.959885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077519.151995:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077519.983901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077520.175871:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077521.007884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077521.199930:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077522.031854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077522.223933:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077522.543852:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077522.543862:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077522.543866:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077522.543874:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3205 00000100:00080000:2.0:1769077522.543883:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077522.543889:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077522.543921:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961443840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077522.543945:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077522.543948:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077522.543949:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077522.543964:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3205 00000100:00080000:2.0:1769077522.543966:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077522.543968:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077522.543976:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077522.543984:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961443968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077522.543991:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077522.544006:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077522.544008:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077522.544011:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3205 00000100:00080000:2.0:1769077522.544014:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077522.544019:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077522.544020:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961443840:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769077522.544023:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077522.544033:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961444096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077522.544091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077522.544124:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077522/real 1769077522] req@ffff895e44e29880 x1855008961443840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077522.544138:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961443840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077522.544146:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077522.544149:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077522.544152:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961443840:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077522.544157:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961443968:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077522.544180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077522.544188:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077522/real 1769077522] req@ffff895e44e28380 x1855008961443968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077522.544196:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961443968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077522.544202:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077522.544204:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077522.544206:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961443968:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077522.544226:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961444096:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077522.544236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077522.544241:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077522/real 1769077522] req@ffff895e44e28000 x1855008961444096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077522.544249:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961444096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077522.544254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077522.544256:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077522.544258:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961444096:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077523.055883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077523.247873:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077524.079934:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077524.271978:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077525.103861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077525.295899:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077526.127877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077526.319906:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077527.151877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077527.343899:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077527.663895:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077527.663912:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077527.663917:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077527.663930:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3210 00000100:00080000:2.0:1769077527.663946:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077527.663955:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077527.664015:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961444224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077527.664052:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077527.664056:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077527.664058:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077527.664063:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3210 00000100:00080000:2.0:1769077527.664067:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077527.664079:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077527.664092:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077527.664105:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961444352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077527.664125:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961444224:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077527.664132:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077527.664139:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077527.664142:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077527.664148:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3210 00000100:00080000:2.0:1769077527.664151:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077527.664154:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077527.664159:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077527.664169:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961444480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077527.664243:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077527.664294:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077527/real 1769077527] req@ffff895e44e2b100 x1855008961444224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077527.664316:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961444224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077527.664328:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077527.664333:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077527.664338:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961444224:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077527.664371:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961444352:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077527.664399:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077527.664414:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077527/real 1769077527] req@ffff895e44e2aa00 x1855008961444352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077527.664429:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961444352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077527.664439:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077527.664443:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077527.664447:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961444352:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077527.664453:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961444480:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077527.664475:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077527.664488:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077527/real 1769077527] req@ffff895e44e2a300 x1855008961444480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077527.664517:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961444480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077527.664526:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077527.664529:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077527.664532:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961444480:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077528.175894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077528.367954:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077529.199871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077529.391849:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077530.223908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077530.416008:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077531.247854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077531.439924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077532.271861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077532.463885:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077532.783881:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077532.783890:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077532.783894:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077532.783902:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3215 00000100:00080000:2.0:1769077532.783913:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077532.783919:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077532.783954:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961444608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077532.783983:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077532.783987:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077532.783990:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077532.783996:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3215 00000100:00080000:2.0:1769077532.784001:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077532.784007:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077532.784031:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077532.784035:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961444608:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077532.784045:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961444736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077532.784055:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077532.784057:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077532.784059:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077532.784062:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3215 00000100:00080000:2.0:1769077532.784066:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077532.784069:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077532.784072:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077532.784082:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961444864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077532.784159:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961444736:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077532.784191:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961444864:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077533.295835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077533.488013:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077534.319816:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077534.511904:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077535.343819:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077535.535845:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769077535.855977:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769077535.856021:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077535.856039:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077535.856079:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077535.856090:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769077535.856136:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077532/real 1769077535] req@ffff895e44e2b480 x1855008961444736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077535.856163:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961444736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077535.856198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077535.856204:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077535.856209:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961444736:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077535.856218:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077532/real 1769077535] req@ffff895e44e28e00 x1855008961444864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077535.856233:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961444864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077535.856242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077535.856246:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077535.856250:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961444864:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077535.856257:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077532/real 1769077535] req@ffff895e44e29f80 x1855008961444608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077535.856268:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961444608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077535.856274:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077535.856277:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077535.856279:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961444608:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077536.367850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077536.559843:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077537.391815:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077537.583914:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077537.903871:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077537.903882:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077537.903886:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077537.903896:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3220 00000100:00080000:3.0:1769077537.903906:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077537.903913:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077537.903945:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961444992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077537.903973:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077537.903977:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077537.903979:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077537.903983:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3220 00000100:00080000:3.0:1769077537.903986:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077537.903989:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077537.903999:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077537.904011:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961445120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077537.904018:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961444992:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077537.904028:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077537.904046:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077537.904048:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077537.904053:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3220 00000100:00080000:3.0:1769077537.904058:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077537.904062:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077537.904067:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077537.904080:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077537.904090:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961445248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077537.904137:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077537/real 1769077537] req@ffff895e453d6300 x1855008961444992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077537.904161:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961444992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077537.904172:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077537.904177:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077537.904180:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961444992:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077537.904206:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961445120:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077537.904248:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077537.904264:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077537/real 1769077537] req@ffff895e453d6d80 x1855008961445120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077537.904279:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961445120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077537.904335:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077537.904339:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077537.904342:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961445120:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077537.904350:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961445248:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077537.904373:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077537.904386:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077537/real 1769077537] req@ffff895e453d4380 x1855008961445248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077537.904397:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961445248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077537.904418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077537.904435:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077537.904439:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961445248:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077538.415852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077538.607910:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077539.439850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077539.632053:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077540.463863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077540.655888:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077541.487859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077541.679909:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077542.511838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077542.703903:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077543.023851:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077543.023859:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077543.023863:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077543.023870:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3225 00000100:00080000:3.0:1769077543.023881:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077543.023887:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077543.023920:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961445376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077543.023945:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077543.023948:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077543.023949:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077543.023952:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3225 00000100:00080000:3.0:1769077543.023954:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077543.023956:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077543.023963:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077543.023971:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961445504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077543.023998:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961445376:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077543.024000:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077543.024006:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077543.024010:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077543.024014:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3225 00000100:00080000:3.0:1769077543.024016:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077543.024019:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077543.024023:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077543.024035:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961445632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077543.024074:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077543.024121:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077543/real 1769077543] req@ffff895e453d6680 x1855008961445376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077543.024140:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961445376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077543.024152:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077543.024164:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077543.024167:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961445376:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077543.024189:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961445504:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077543.024208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077543.024218:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077543/real 1769077543] req@ffff895e453d5180 x1855008961445504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077543.024236:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961445504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077543.024257:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077543.024262:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077543.024266:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961445504:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077543.024273:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961445632:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077543.024296:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077543.024308:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077543/real 1769077543] req@ffff895e453d5880 x1855008961445632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077543.024321:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961445632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077543.024330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077543.024333:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077543.024368:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961445632:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077543.535874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077543.727883:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077544.559822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077544.751919:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077545.583831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077545.775903:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077546.607886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077546.799879:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077547.631938:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077547.823977:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077548.143884:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077548.143896:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077548.143901:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077548.143911:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3230 00000100:00080000:3.0:1769077548.143924:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077548.143931:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077548.143976:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961445760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077548.144008:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077548.144011:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077548.144012:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077548.144016:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3230 00000100:00080000:3.0:1769077548.144018:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077548.144021:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077548.144049:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077548.144059:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961445888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077548.144068:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077548.144071:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077548.144072:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077548.144075:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3230 00000100:00080000:3.0:1769077548.144078:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077548.144081:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077548.144093:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077548.144098:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961445760:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077548.144104:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961446016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077548.144178:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077548.144230:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077548/real 1769077548] req@ffff895e453d4700 x1855008961445760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077548.144245:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961445760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077548.144289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077548.144293:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077548.144297:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961445760:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077548.144303:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961445888:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077548.144323:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077548.144332:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077548/real 1769077548] req@ffff895e453d7100 x1855008961445888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077548.144341:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961445888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077548.144347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077548.144350:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077548.144352:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961445888:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077548.144374:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961446016:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077548.144385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077548.144390:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077548/real 1769077548] req@ffff895e453d7b80 x1855008961446016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077548.144398:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961446016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077548.144404:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077548.144414:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077548.144417:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961446016:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077548.655894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077548.847848:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077549.679964:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077549.871900:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077550.703822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077550.895963:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077551.727883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077551.919955:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077552.751870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077552.944022:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077553.263867:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077553.263876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077553.263881:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077553.263889:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3235 00000100:00080000:0.0:1769077553.263897:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077553.263903:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077553.263935:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961446144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077553.263959:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077553.263961:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077553.263963:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077553.263965:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3235 00000100:00080000:0.0:1769077553.263968:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077553.263971:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077553.263980:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077553.263992:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961446144:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077553.263997:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961446272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077553.264011:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077553.264015:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077553.264017:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077553.264022:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3235 00000100:00080000:0.0:1769077553.264027:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077553.264031:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:3.0:1769077553.264049:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:0.0:1769077553.264054:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077553.264070:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961446400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077553.264098:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077553/real 1769077553] req@ffff895e4383df80 x1855008961446144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077553.264114:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961446144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077553.264122:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077553.264125:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077553.264128:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961446144:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077553.264147:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961446272:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077553.264198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077553.264208:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077553/real 1769077553] req@ffff895e4383ed80 x1855008961446272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077553.264217:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961446272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077553.264224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077553.264226:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077553.264229:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961446272:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077553.264233:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961446400:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077553.264246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077553.264253:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077553/real 1769077553] req@ffff895e4383d180 x1855008961446400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077553.264260:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961446400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077553.264265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077553.264267:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077553.264269:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961446400:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077553.775858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077553.967873:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077554.799808:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077554.991917:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077555.823888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077556.015903:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077556.847841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077557.039897:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077557.871883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077558.063941:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077558.383882:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077558.383891:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077558.383895:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077558.383903:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3240 00000100:00080000:1.0:1769077558.383913:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077558.383919:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077558.383951:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961446528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077558.383972:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077558.383974:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077558.383976:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077558.383978:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3240 00000100:00080000:1.0:1769077558.383981:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077558.383983:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077558.383990:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077558.384012:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961446656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077558.384020:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077558.384022:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077558.384024:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077558.384027:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3240 00000100:00080000:1.0:1769077558.384030:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077558.384033:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077558.384035:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077558.384042:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961446784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077558.384052:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961446528:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769077558.384130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077558.384168:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077558/real 1769077558] req@ffff895e4713f480 x1855008961446528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077558.384183:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961446528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077558.384190:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077558.384194:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077558.384197:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961446528:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077558.384203:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961446656:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077558.384218:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077558.384227:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077558/real 1769077558] req@ffff895e4713ce00 x1855008961446656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077558.384240:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961446656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077558.384248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077558.384251:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077558.384256:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961446656:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077558.384280:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961446784:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077558.384300:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077558.384312:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077558/real 1769077558] req@ffff895e4713d500 x1855008961446784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077558.384338:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961446784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077558.384346:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077558.384348:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077558.384352:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961446784:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077558.895835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077559.087923:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077559.919851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077560.111897:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077560.943847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077561.135911:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077561.967870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077562.159928:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077562.991846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077563.183922:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077563.503864:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077563.503873:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077563.503876:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077563.503884:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3246 00000100:00080000:2.0:1769077563.503894:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077563.503933:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077563.503961:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961446912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077563.504002:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077563.504007:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077563.504009:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077563.504013:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3246 00000100:00080000:2.0:1769077563.504018:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077563.504023:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077563.504049:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077563.504056:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961446912:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077563.504067:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961447040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077563.504079:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077563.504082:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077563.504085:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077563.504089:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3246 00000100:00080000:2.0:1769077563.504094:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077563.504099:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077563.504104:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077563.504121:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961447168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077563.504121:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077563.504174:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077563/real 1769077563] req@ffff895e44e29500 x1855008961446912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077563.504189:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961446912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077563.504197:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077563.504216:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077563.504218:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961446912:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077563.504241:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961447040:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077563.504265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077563.504277:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077563/real 1769077563] req@ffff895e44e28a80 x1855008961447040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077563.504300:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961447040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077563.504308:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077563.504311:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077563.504315:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961447040:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077563.504322:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961447168:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077563.504343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077563.504355:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077563/real 1769077563] req@ffff895e44e29180 x1855008961447168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077563.504367:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961447168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077563.504376:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077563.504379:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077563.504383:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961447168:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077564.015886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077564.208075:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077565.039881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077565.232091:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077566.063814:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077566.255908:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077567.087843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077567.279879:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077568.111837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077568.303856:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077568.623873:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077568.623927:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077568.623932:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077568.623943:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3251 00000100:00080000:2.0:1769077568.623956:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077568.623970:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077568.624021:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961447296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077568.624060:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077568.624063:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077568.624065:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077568.624068:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3251 00000100:00080000:2.0:1769077568.624072:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077568.624076:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077568.624086:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077568.624101:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961447424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077568.624114:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077568.624119:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077568.624124:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077568.624124:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961447296:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077568.624135:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3251 00000100:00080000:2.0:1769077568.624140:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077568.624145:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077568.624152:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077568.624167:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961447552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077568.624220:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077568.624276:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077568/real 1769077568] req@ffff895e44e2bb80 x1855008961447296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077568.624309:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961447296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077568.624320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077568.624324:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077568.624327:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961447296:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077568.624356:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961447424:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077568.624375:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077568.624385:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077568/real 1769077568] req@ffff895e44e2b800 x1855008961447424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077568.624393:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961447424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077568.624400:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077568.624402:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077568.624405:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961447424:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077568.624408:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961447552:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077568.624419:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077568.624425:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077568/real 1769077568] req@ffff895e44e29c00 x1855008961447552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077568.624433:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961447552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077568.624448:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077568.624451:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077568.624453:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961447552:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077569.135937:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077569.327981:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077570.159831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077570.351938:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077571.183888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077571.375918:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077572.207826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077572.399869:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077573.231914:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077573.423972:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077573.743956:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077573.743979:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077573.743984:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077573.743995:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3256 00000100:00080000:2.0:1769077573.744007:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077573.744015:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077573.744062:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961447680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077573.744091:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077573.744094:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077573.744096:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077573.744099:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3256 00000100:00080000:2.0:1769077573.744101:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077573.744104:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077573.744113:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077573.744122:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961447808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077573.744130:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077573.744133:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077573.744134:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077573.744137:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3256 00000100:00100000:3.0:1769077573.744153:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961447680:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077573.744158:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077573.744166:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077573.744170:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077573.744180:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961447936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077573.744247:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077573.744293:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077573/real 1769077573] req@ffff895e44e29c00 x1855008961447680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077573.744311:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961447680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077573.744320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077573.744324:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077573.744327:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961447680:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077573.744332:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961447808:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077573.744350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077573.744358:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077573/real 1769077573] req@ffff895e44e2b800 x1855008961447808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077573.744370:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961447808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077573.744389:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077573.744394:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077573.744398:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961447808:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077573.744431:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961447936:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077573.744455:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077573.744466:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077573/real 1769077573] req@ffff895e44e2bb80 x1855008961447936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077573.744478:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961447936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077573.744484:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077573.744487:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077573.744489:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961447936:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077574.255828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077574.447912:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077575.279933:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077575.471934:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077576.303879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077576.495899:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077577.327877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077577.519879:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077578.351839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077578.543856:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077578.863865:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077578.863889:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077578.863893:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077578.863900:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3261 00000100:00080000:1.0:1769077578.863909:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077578.863915:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077578.863944:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961448064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077578.863967:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077578.863970:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077578.863971:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077578.863974:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3261 00000100:00080000:1.0:1769077578.863976:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077578.863979:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077578.863985:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077578.863996:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961448192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077578.864005:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077578.864008:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077578.864009:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077578.864014:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3261 00000100:00080000:1.0:1769077578.864016:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077578.864016:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961448064:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077578.864023:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077578.864027:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077578.864036:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961448320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077578.864078:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077578.864127:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077578/real 1769077578] req@ffff895e4713ea00 x1855008961448064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077578.864148:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961448064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077578.864172:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077578.864185:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077578.864190:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961448064:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077578.864216:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961448192:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077578.864259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077578.864275:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077578/real 1769077578] req@ffff895e4713f100 x1855008961448192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077578.864288:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961448192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077578.864296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077578.864300:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077578.864303:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961448192:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077578.864309:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961448320:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077578.864330:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077578.864342:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077578/real 1769077578] req@ffff895e4713e300 x1855008961448320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077578.864355:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961448320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077578.864364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077578.864367:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077578.864370:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961448320:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077579.375865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077579.567930:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077580.399826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077580.591919:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077581.423851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077581.615887:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077582.447863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077582.639914:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077583.471817:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077583.663881:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077583.983873:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077583.983885:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077583.983890:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077583.983900:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3266 00000100:00080000:2.0:1769077583.983910:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077583.983917:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077583.983956:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961448448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077583.983985:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077583.983988:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077583.983994:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077583.983998:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3266 00000100:00080000:2.0:1769077583.984002:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077583.984005:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077583.984015:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077583.984029:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961448576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077583.984039:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961448448:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077583.984059:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077583.984063:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077583.984065:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077583.984071:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3266 00000100:00080000:2.0:1769077583.984075:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077583.984080:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077583.984086:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077583.984100:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961448704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077583.984109:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077583.984147:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077583/real 1769077583] req@ffff895e44e29180 x1855008961448448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077583.984163:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961448448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077583.984171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077583.984174:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077583.984177:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961448448:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077583.984197:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961448576:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077583.984211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077583.984219:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077583/real 1769077583] req@ffff895e44e28a80 x1855008961448576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077583.984227:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961448576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077583.984233:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077583.984235:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077583.984237:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961448576:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077583.984247:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961448704:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077583.984260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077583.984266:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077583/real 1769077583] req@ffff895e44e29500 x1855008961448704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077583.984274:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961448704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077583.984279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077583.984280:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077583.984282:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961448704:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077584.495861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077584.687893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077585.519821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077585.711873:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077586.543857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077586.735903:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077587.567862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077587.759868:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077588.591852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077588.783883:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077589.103833:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077589.103841:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077589.103845:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077589.103852:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3271 00000100:00080000:3.0:1769077589.103863:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077589.103869:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077589.103900:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961448832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077589.103924:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077589.103927:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077589.103928:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077589.103931:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3271 00000100:00080000:3.0:1769077589.103948:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077589.103951:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077589.103959:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077589.103966:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961448832:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077589.103971:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961448960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077589.103981:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077589.103983:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077589.103985:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077589.103987:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3271 00000100:00080000:3.0:1769077589.103990:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077589.103992:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077589.103996:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077589.104003:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961449088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077589.104023:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077589.104054:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077589/real 1769077589] req@ffff895e453d5500 x1855008961448832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077589.104068:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961448832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077589.104076:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077589.104079:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077589.104082:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961448832:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077589.104101:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961448960:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077589.104122:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077589.104136:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077589/real 1769077589] req@ffff895e453d7480 x1855008961448960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077589.104150:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961448960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077589.104159:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077589.104163:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077589.104173:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961448960:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077589.104180:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961449088:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077589.104204:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077589.104217:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077589/real 1769077589] req@ffff895e453d4e00 x1855008961449088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077589.104241:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961449088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077589.104251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077589.104254:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077589.104258:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961449088:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077589.615830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077589.807918:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077590.639842:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077590.831901:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077591.663821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077591.855878:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077592.687852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077592.879927:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077593.711833:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077593.903913:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077594.223877:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077594.223889:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077594.223893:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077594.223903:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3276 00000100:00080000:3.0:1769077594.223914:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077594.223921:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077594.223959:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961449216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077594.223987:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077594.224005:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077594.224008:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077594.224013:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3276 00000100:00080000:3.0:1769077594.224020:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077594.224026:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961449216:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077594.224028:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077594.224051:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077594.224068:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961449344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077594.224081:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077594.224084:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077594.224087:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077594.224091:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3276 00000100:00080000:3.0:1769077594.224095:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077594.224100:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:0.0:1769077594.224106:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:3.0:1769077594.224108:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077594.224124:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961449472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077594.224149:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077594/real 1769077594] req@ffff895e453d4a80 x1855008961449216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077594.224169:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961449216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077594.224179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077594.224188:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077594.224199:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961449216:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077594.224220:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961449344:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077594.224242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077594.224253:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077594/real 1769077594] req@ffff895e453d5c00 x1855008961449344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077594.224278:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961449344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077594.224286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077594.224289:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077594.224293:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961449344:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077594.224299:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961449472:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077594.224316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077594.224326:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077594/real 1769077594] req@ffff895e453d7800 x1855008961449472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077594.224338:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961449472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077594.224345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077594.224349:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077594.224352:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961449472:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077594.735862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077594.927923:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077595.759904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077595.951969:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077596.783853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077596.975907:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077597.807870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077597.999953:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077598.831894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077599.023873:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077599.343891:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077599.343918:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077599.343921:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077599.343935:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3281 00000100:00080000:0.0:1769077599.343945:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077599.343951:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077599.343982:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961449600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077599.344006:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077599.344009:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077599.344010:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077599.344013:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3281 00000100:00080000:0.0:1769077599.344015:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077599.344018:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077599.344026:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077599.344036:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961449728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077599.344044:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961449600:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077599.344052:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077599.344056:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077599.344057:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077599.344061:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3281 00000100:00080000:0.0:1769077599.344063:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077599.344066:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077599.344070:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077599.344079:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961449856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077599.344151:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961449728:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077599.344176:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961449856:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077599.855861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077600.047917:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077600.879962:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077601.071920:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077601.903885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077602.095956:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769077602.415953:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769077602.415990:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077602.416013:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769077602.416047:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769077602.416056:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769077602.416104:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077599/real 1769077602] req@ffff895e4383ed80 x1855008961449728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077602.416124:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961449728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077602.416134:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077602.416138:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077602.416152:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961449728:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077602.416158:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077599/real 1769077602] req@ffff895e4383df80 x1855008961449856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077602.416166:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961449856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077602.416171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077602.416175:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077602.416178:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961449856:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077602.416184:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077599/real 1769077602] req@ffff895e4383d180 x1855008961449600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077602.416198:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961449600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077602.416206:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077602.416209:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077602.416213:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961449600:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077602.927883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077603.119892:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077603.951876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077604.143952:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077604.463877:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077604.463885:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077604.463889:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077604.463897:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3287 00000100:00080000:3.0:1769077604.463907:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077604.463913:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077604.463944:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961449984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077604.463974:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077604.463978:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077604.463980:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077604.463984:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3287 00000100:00080000:3.0:1769077604.463987:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077604.463991:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077604.464001:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077604.464020:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961450112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077604.464022:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961449984:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077604.464042:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077604.464046:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077604.464049:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077604.464054:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3287 00000100:00080000:3.0:1769077604.464058:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077604.464064:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077604.464086:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077604.464094:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077604.464104:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961450240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077604.464136:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077604/real 1769077604] req@ffff895e453d5f80 x1855008961449984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077604.464157:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961449984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077604.464168:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077604.464173:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077604.464177:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961449984:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077604.464201:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961450112:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077604.464222:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077604.464233:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077604/real 1769077604] req@ffff895e453d7800 x1855008961450112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077604.464246:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961450112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077604.464255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077604.464259:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077604.464262:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961450112:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077604.464269:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961450240:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077604.464285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077604.464336:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077604/real 1769077604] req@ffff895e453d5c00 x1855008961450240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077604.464350:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961450240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077604.464359:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077604.464362:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077604.464367:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961450240:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077604.975917:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077605.167905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077605.999862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077606.191978:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077607.023896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077607.216007:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077608.047920:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077608.239939:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077609.071898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077609.264048:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077609.583912:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077609.583924:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077609.583952:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077609.583963:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3292 00000100:00080000:2.0:1769077609.583987:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077609.583996:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077609.584046:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961450368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077609.584078:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077609.584080:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077609.584082:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077609.584085:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3292 00000100:00080000:2.0:1769077609.584088:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077609.584092:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077609.584104:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077609.584120:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961450496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077609.584130:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077609.584134:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077609.584136:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077609.584142:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3292 00000100:00080000:2.0:1769077609.584148:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077609.584160:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961450368:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077609.584160:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077609.584169:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077609.584186:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961450624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077609.584259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077609.584308:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077609/real 1769077609] req@ffff895e44e29f80 x1855008961450368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077609.584330:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961450368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077609.584342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077609.584346:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077609.584351:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961450368:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077609.584360:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961450496:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077609.584386:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077609.584399:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077609/real 1769077609] req@ffff895e44e28e00 x1855008961450496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077609.584413:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961450496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077609.584442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077609.584446:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077609.584450:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961450496:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077609.584482:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961450624:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077609.584507:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077609.584520:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077609/real 1769077609] req@ffff895e44e2b480 x1855008961450624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077609.584532:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961450624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077609.584540:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077609.584544:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077609.584548:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961450624:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077610.095885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077610.287934:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077611.119848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077611.311943:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077612.143921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077612.335995:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077613.167935:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077613.359929:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077614.191853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077614.383928:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077614.703864:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077614.703874:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077614.703878:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077614.703885:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3297 00000100:00080000:2.0:1769077614.703895:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077614.703901:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077614.703932:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961450752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077614.703955:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077614.703957:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077614.703958:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077614.703961:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3297 00000100:00080000:2.0:1769077614.703964:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077614.703966:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077614.703973:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077614.703982:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961450880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077614.703989:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077614.703992:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077614.703993:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077614.703996:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3297 00000100:00080000:2.0:1769077614.703998:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077614.704015:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077614.704018:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077614.704025:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961450752:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077614.704030:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961451008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077614.704097:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077614.704166:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077614/real 1769077614] req@ffff895e44e2a300 x1855008961450752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077614.704185:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961450752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077614.704195:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077614.704200:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077614.704215:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961450752:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077614.704223:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961450880:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077614.704247:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077614.704261:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077614/real 1769077614] req@ffff895e44e2aa00 x1855008961450880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077614.704275:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961450880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077614.704283:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077614.704286:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077614.704290:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961450880:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077614.704316:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961451008:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077614.704336:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077614.704347:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077614/real 1769077614] req@ffff895e44e2b100 x1855008961451008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077614.704358:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961451008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077614.704366:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077614.704369:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077614.704373:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961451008:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077615.215869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077615.407977:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077616.239878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077616.431940:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077617.263907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077617.455922:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077618.287864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077618.479885:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077619.311848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077619.503912:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077619.823836:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077619.823845:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077619.823863:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077619.823872:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3302 00000100:00080000:0.0:1769077619.823892:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077619.823900:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077619.823935:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961451136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077619.823960:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077619.823963:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077619.823964:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077619.823967:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3302 00000100:00080000:0.0:1769077619.823970:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077619.823974:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077619.823982:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077619.823991:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961451264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077619.823996:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961451136:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077619.824005:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077619.824008:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077619.824009:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077619.824012:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3302 00000100:00080000:0.0:1769077619.824014:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077619.824017:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077619.824020:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077619.824035:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961451392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077619.824084:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077619.824132:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077619/real 1769077619] req@ffff895e4383d880 x1855008961451136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077619.824150:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961451136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077619.824158:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077619.824161:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077619.824164:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961451136:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077619.824193:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961451264:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077619.824219:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077619.824233:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077619/real 1769077619] req@ffff895e4383e680 x1855008961451264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077619.824254:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961451264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077619.824263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077619.824265:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077619.824268:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961451264:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077619.824273:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961451392:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077619.824297:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077619.824305:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077619/real 1769077619] req@ffff895e4383c700 x1855008961451392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077619.824313:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961451392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077619.824318:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077619.824320:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077619.824322:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961451392:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077620.335839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077620.527886:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077621.359867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077621.551983:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077622.383867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077622.575963:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077623.407911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077623.599967:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077624.431877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077624.623898:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077624.943928:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077624.943938:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077624.943941:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077624.943953:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3307 00000100:00080000:2.0:1769077624.943982:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077624.943988:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077624.944027:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961451520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077624.944051:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077624.944054:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077624.944055:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077624.944058:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3307 00000100:00080000:2.0:1769077624.944061:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077624.944063:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077624.944070:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077624.944084:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961451648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077624.944106:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961451520:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077624.944112:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077624.944116:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077624.944118:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077624.944122:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3307 00000100:00080000:2.0:1769077624.944125:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077624.944127:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077624.944131:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077624.944141:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961451776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077624.944176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077624.944226:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077624/real 1769077624] req@ffff895e44e28000 x1855008961451520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077624.944248:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961451520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077624.944261:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077624.944265:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077624.944270:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961451520:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077624.944295:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961451648:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077624.944320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077624.944334:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077624/real 1769077624] req@ffff895e44e28380 x1855008961451648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077624.944347:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961451648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077624.944357:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077624.944361:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077624.944365:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961451648:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077624.944372:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961451776:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077624.944392:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077624.944403:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077624/real 1769077624] req@ffff895e44e29880 x1855008961451776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077624.944425:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961451776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077624.944433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077624.944437:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077624.944441:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961451776:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077625.455845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077625.647936:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077626.479827:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077626.671896:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077627.503837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077627.695945:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077628.527827:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077628.719914:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077629.551853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077629.743934:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077630.063926:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077630.063936:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077630.063941:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077630.063952:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3312 00000100:00080000:1.0:1769077630.063962:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077630.063969:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077630.064007:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961451904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077630.064039:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077630.064043:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077630.064045:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077630.064050:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3312 00000100:00080000:1.0:1769077630.064054:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077630.064058:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077630.064068:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077630.064085:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961452032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077630.064086:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961451904:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077630.064126:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077630.064129:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077630.064131:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077630.064136:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3312 00000100:00080000:1.0:1769077630.064142:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077630.064149:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:3.0:1769077630.064150:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:1.0:1769077630.064159:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077630.064175:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961452160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077630.064188:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077630/real 1769077630] req@ffff895e4713d880 x1855008961451904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077630.064211:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961451904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077630.064219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077630.064224:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077630.064228:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961451904:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077630.064249:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961452032:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077630.064264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077630.064273:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077630/real 1769077630] req@ffff895e4713ed80 x1855008961452032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077630.064289:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961452032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077630.064296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077630.064298:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077630.064300:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961452032:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077630.064304:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961452160:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077630.064318:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077630.064324:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077630/real 1769077630] req@ffff895e4713c000 x1855008961452160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077630.064332:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961452160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077630.064337:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077630.064338:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077630.064340:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961452160:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077630.575827:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077630.767914:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077631.599835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077631.791914:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077632.623827:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077632.815921:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077633.647861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077633.839928:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077634.671899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077634.863944:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077635.183902:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077635.183914:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077635.183918:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077635.183929:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3317 00000100:00080000:0.0:1769077635.183939:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077635.183946:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077635.183982:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961452288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077635.184007:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077635.184010:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077635.184012:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077635.184014:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3317 00000100:00080000:0.0:1769077635.184017:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077635.184020:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077635.184028:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077635.184038:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961452416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077635.184045:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961452288:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077635.184049:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077635.184053:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077635.184055:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077635.184057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3317 00000100:00080000:0.0:1769077635.184059:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077635.184062:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077635.184081:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077635.184094:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961452544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077635.184104:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077635.184146:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077635/real 1769077635] req@ffff895e4383fb80 x1855008961452288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077635.184169:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961452288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077635.184177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077635.184181:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077635.184184:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961452288:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077635.184203:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961452416:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077635.184219:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077635.184228:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077635/real 1769077635] req@ffff895e4383f100 x1855008961452416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077635.184236:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961452416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077635.184242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077635.184245:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077635.184247:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961452416:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077635.184250:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961452544:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077635.184261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077635.184266:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077635/real 1769077635] req@ffff895e4383c380 x1855008961452544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077635.184273:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961452544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077635.184288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077635.184290:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077635.184293:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961452544:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077635.695873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077635.887971:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077636.719902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077636.911971:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077637.743888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077637.935952:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077638.767853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077638.959911:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077639.791868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077639.983897:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077640.303909:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077640.303917:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077640.303921:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077640.303928:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3322 00000100:00080000:1.0:1769077640.303946:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077640.303952:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077640.303986:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961452672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077640.304011:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077640.304029:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077640.304030:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077640.304034:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3322 00000100:00080000:1.0:1769077640.304039:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077640.304043:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077640.304052:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077640.304063:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961452800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077640.304077:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077640.304078:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961452672:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077640.304086:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077640.304093:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077640.304099:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3322 00000100:00080000:1.0:1769077640.304101:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077640.304104:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077640.304108:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077640.304118:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961452928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077640.304169:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077640.304215:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077640/real 1769077640] req@ffff895e4713c700 x1855008961452672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077640.304238:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961452672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077640.304251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077640.304257:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077640.304261:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961452672:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077640.304290:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961452800:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077640.304310:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077640.304320:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077640/real 1769077640] req@ffff895e4713fb80 x1855008961452800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077640.304328:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961452800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077640.304346:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077640.304349:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077640.304351:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961452800:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077640.304356:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961452928:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077640.304371:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077640.304378:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077640/real 1769077640] req@ffff895e4713c000 x1855008961452928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077640.304386:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961452928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077640.304392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077640.304394:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077640.304396:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961452928:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077640.815877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077641.007918:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077641.839854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077642.031871:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077642.863864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077643.055935:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077643.887830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077644.079880:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077644.911881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077645.103912:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077645.423877:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077645.423886:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077645.423890:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077645.423897:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3327 00000100:00080000:3.0:1769077645.423907:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077645.423913:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077645.423970:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961453056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077645.423995:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077645.423998:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077645.423999:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077645.424002:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3327 00000100:00080000:3.0:1769077645.424005:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077645.424007:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077645.424015:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077645.424023:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961453184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077645.424032:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077645.424035:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077645.424038:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077645.424039:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961453056:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077645.424043:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3327 00000100:00080000:3.0:1769077645.424048:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077645.424052:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077645.424055:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077645.424075:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961453312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077645.424120:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077645.424161:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077645/real 1769077645] req@ffff895e453d4a80 x1855008961453056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077645.424177:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961453056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077645.424184:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077645.424187:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077645.424190:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961453056:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077645.424210:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961453184:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077645.424228:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077645.424236:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077645/real 1769077645] req@ffff895e453d4e00 x1855008961453184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077645.424254:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961453184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077645.424259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077645.424261:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077645.424263:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961453184:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077645.424267:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961453312:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077645.424288:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077645.424295:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077645/real 1769077645] req@ffff895e453d7480 x1855008961453312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077645.424303:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961453312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077645.424309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077645.424311:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077645.424313:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961453312:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077645.935865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077646.127874:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077646.959888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077647.151900:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077647.983851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077648.175929:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077649.007871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077649.199901:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077650.031826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077650.223925:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077650.543859:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077650.543876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077650.543881:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077650.543892:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3333 00000100:00080000:2.0:1769077650.543903:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077650.543911:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077650.543950:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961453440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077650.544001:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077650.544005:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077650.544007:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077650.544012:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3333 00000100:00080000:2.0:1769077650.544017:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077650.544021:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077650.544031:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077650.544046:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961453568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077650.544057:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077650.544061:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077650.544065:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077650.544067:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961453440:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077650.544072:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3333 00000100:00080000:2.0:1769077650.544079:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077650.544083:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077650.544088:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077650.544100:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961453696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077650.544148:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077650.544185:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077650/real 1769077650] req@ffff895e44e29880 x1855008961453440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077650.544204:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961453440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077650.544216:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077650.544220:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077650.544224:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961453440:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077650.544249:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961453568:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077650.544282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077650.544294:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077650/real 1769077650] req@ffff895e44e28380 x1855008961453568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077650.544324:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961453568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077650.544333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077650.544337:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077650.544341:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961453568:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077650.544348:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961453696:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077650.544380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077650.544393:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077650/real 1769077650] req@ffff895e44e28000 x1855008961453696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077650.544406:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961453696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077650.544415:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077650.544418:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077650.544422:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961453696:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077651.055853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077651.247924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077652.079862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077652.272008:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077653.103883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077653.295957:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077654.127877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077654.319956:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077655.151859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077655.343938:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077655.663924:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077655.663939:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077655.663944:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077655.663956:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3338 00000100:00080000:0.0:1769077655.663971:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077655.663980:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077655.664030:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961453824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077655.664090:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077655.664095:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077655.664097:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077655.664102:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3338 00000100:00080000:0.0:1769077655.664107:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077655.664113:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077655.664124:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077655.664143:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961453824:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077655.664144:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961453952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077655.664160:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077655.664165:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077655.664168:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077655.664173:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3338 00000100:00080000:0.0:1769077655.664177:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077655.664182:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077655.664188:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077655.664201:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961454080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077655.664228:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077655.664281:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077655/real 1769077655] req@ffff895e4383ce00 x1855008961453824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077655.664302:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961453824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077655.664312:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077655.664316:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077655.664319:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961453824:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077655.664343:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961453952:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077655.664363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077655.664372:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077655/real 1769077655] req@ffff895e4383f480 x1855008961453952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077655.664381:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961453952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077655.664387:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077655.664389:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077655.664392:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961453952:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077655.664395:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961454080:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077655.664406:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077655.664412:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077655/real 1769077655] req@ffff895e4383c000 x1855008961454080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077655.664432:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961454080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077655.664438:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077655.664441:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077655.664457:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961454080:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077656.175986:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077656.367960:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077657.199864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077657.391901:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077658.223914:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077658.415972:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077659.247919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077659.439883:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077660.271859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077660.463956:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077660.783835:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077660.783848:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077660.783852:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077660.783864:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3343 00000100:00080000:1.0:1769077660.783875:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077660.783883:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077660.783928:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961454208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077660.783955:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077660.783958:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077660.783959:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077660.783962:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3343 00000100:00080000:1.0:1769077660.783965:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077660.783967:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077660.783977:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077660.784007:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961454336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077660.784020:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961454208:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077660.784023:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077660.784035:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077660.784039:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077660.784043:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3343 00000100:00080000:1.0:1769077660.784046:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077660.784049:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077660.784052:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077660.784063:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961454464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077660.784111:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077660.784151:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077660/real 1769077660] req@ffff895e4713ed80 x1855008961454208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077660.784171:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961454208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077660.784182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077660.784187:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077660.784192:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961454208:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077660.784217:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961454336:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077660.784238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077660.784250:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077660/real 1769077660] req@ffff895e4713d880 x1855008961454336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077660.784262:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961454336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077660.784289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077660.784293:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077660.784297:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961454336:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077660.784303:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961454464:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077660.784323:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077660.784333:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077660/real 1769077660] req@ffff895e4713e300 x1855008961454464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077660.784345:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961454464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077660.784353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077660.784372:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077660.784376:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961454464:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077661.295867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077661.487910:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077662.319852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077662.511905:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077663.343855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077663.535858:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077664.367857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077664.559844:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077665.391821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077665.583879:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077665.903867:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077665.903875:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077665.903879:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077665.903888:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3348 00000100:00080000:2.0:1769077665.903897:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077665.903903:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077665.903935:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961454592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077665.903964:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077665.903968:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077665.903970:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077665.903974:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3348 00000100:00080000:2.0:1769077665.903978:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077665.903982:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077665.903991:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077665.904004:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961454720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077665.904014:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077665.904018:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077665.904036:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077665.904040:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3348 00000100:00080000:2.0:1769077665.904043:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077665.904046:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077665.904051:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077665.904059:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961454848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077665.904071:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961454592:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077665.904176:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961454720:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077665.904218:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961454848:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077666.415848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077666.607845:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077667.439819:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077667.631878:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077668.463858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077668.655902:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769077668.976003:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769077668.976047:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077668.976062:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077668.976095:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077668.976105:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769077668.976152:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077665/real 1769077668] req@ffff895e44e2a300 x1855008961454848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077668.976177:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961454848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077668.976208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077668.976213:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077668.976217:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961454848:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077668.976239:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077665/real 1769077668] req@ffff895e44e2b100 x1855008961454592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077668.976264:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961454592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077668.976273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077668.976278:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077668.976281:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961454592:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077668.976290:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077665/real 1769077668] req@ffff895e44e2aa00 x1855008961454720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077668.976305:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961454720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077668.976315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077668.976320:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077668.976328:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961454720:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077669.487814:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077669.679869:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077670.511832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077670.703849:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077671.023905:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077671.023915:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077671.023918:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077671.023926:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3353 00000100:00080000:0.0:1769077671.023935:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077671.023941:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077671.023972:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961454976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077671.023997:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077671.023999:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077671.024001:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077671.024004:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3353 00000100:00080000:0.0:1769077671.024006:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077671.024009:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077671.024015:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077671.024024:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961455104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077671.024034:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077671.024037:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077671.024040:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077671.024044:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3353 00000100:00080000:0.0:1769077671.024047:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077671.024056:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961454976:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077671.024070:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077671.024076:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077671.024089:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961455232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077671.024144:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077671.024216:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077671/real 1769077671] req@ffff895e4383ea00 x1855008961454976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077671.024239:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961454976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077671.024250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077671.024255:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077671.024260:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961454976:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077671.024269:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961455104:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077671.024305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077671.024319:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077671/real 1769077671] req@ffff895e4383dc00 x1855008961455104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077671.024332:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961455104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077671.024341:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077671.024345:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077671.024348:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961455104:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077671.024376:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961455232:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077671.024398:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077671.024410:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077671/real 1769077671] req@ffff895e4383ca80 x1855008961455232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077671.024424:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961455232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077671.024456:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077671.024460:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077671.024464:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961455232:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077671.535868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077671.727905:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077672.559866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077672.751868:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077673.583831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077673.775919:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077674.607820:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077674.799866:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077675.631854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077675.823859:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077676.143835:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077676.143843:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077676.143847:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077676.143855:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3358 00000100:00080000:0.0:1769077676.143865:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077676.143870:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077676.143902:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961455360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077676.143925:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077676.143927:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077676.143929:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077676.143931:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3358 00000100:00080000:0.0:1769077676.143933:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077676.143936:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077676.143943:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077676.143951:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961455488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077676.143975:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077676.143978:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769077676.143978:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961455360:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077676.143981:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077676.143989:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3358 00000100:00080000:0.0:1769077676.143992:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077676.143995:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077676.143999:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077676.144010:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961455616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077676.144052:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077676.144093:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077676/real 1769077676] req@ffff895e4383f800 x1855008961455360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077676.144129:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961455360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077676.144141:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077676.144145:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077676.144150:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961455360:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077676.144174:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961455488:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077676.144200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077676.144215:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077676/real 1769077676] req@ffff895e4383e300 x1855008961455488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077676.144229:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961455488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077676.144237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077676.144251:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077676.144255:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961455488:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077676.144263:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961455616:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077676.144287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077676.144300:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077676/real 1769077676] req@ffff895e4383d500 x1855008961455616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077676.144313:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961455616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077676.144321:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077676.144325:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077676.144328:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961455616:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077676.655849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077676.847890:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077677.679871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077677.871924:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077678.703889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077678.895885:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077679.727853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077679.919915:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077680.751863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077680.943857:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077681.263929:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077681.263937:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077681.263942:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077681.263949:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3363 00000100:00080000:1.0:1769077681.263967:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077681.263973:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077681.264004:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961455744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077681.264051:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077681.264054:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077681.264056:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077681.264059:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3363 00000100:00080000:1.0:1769077681.264062:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077681.264065:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077681.264072:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077681.264081:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961455872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077681.264089:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077681.264092:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077681.264093:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077681.264095:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3363 00000100:00080000:1.0:1769077681.264097:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077681.264111:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961455744:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077681.264117:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077681.264125:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077681.264146:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961456000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077681.264196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077681.264237:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077681/real 1769077681] req@ffff895e4713f100 x1855008961455744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077681.264258:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961455744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077681.264269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077681.264274:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077681.264278:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961455744:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077681.264286:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961455872:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077681.264319:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077681.264333:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077681/real 1769077681] req@ffff895e4713ea00 x1855008961455872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077681.264347:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961455872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077681.264356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077681.264360:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077681.264364:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961455872:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077681.264388:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961456000:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077681.264413:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077681.264424:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077681/real 1769077681] req@ffff895e4713d500 x1855008961456000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077681.264436:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961456000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077681.264444:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077681.264448:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077681.264451:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961456000:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077681.775839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077681.967880:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077682.799850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077682.991871:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077683.823857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077684.015894:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077684.847901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077685.039931:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077685.871849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077686.063968:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077686.383878:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077686.383886:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077686.383890:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077686.383897:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3368 00000100:00080000:0.0:1769077686.383915:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077686.383923:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077686.383975:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961456128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077686.384001:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077686.384003:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077686.384005:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077686.384027:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3368 00000100:00080000:0.0:1769077686.384031:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077686.384035:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077686.384043:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077686.384055:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961456256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077686.384059:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961456128:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077686.384066:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077686.384070:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077686.384072:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077686.384075:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3368 00000100:00080000:0.0:1769077686.384077:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077686.384079:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077686.384082:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077686.384090:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961456384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077686.384129:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077686.384170:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077686/real 1769077686] req@ffff895e4383d500 x1855008961456128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077686.384197:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961456128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077686.384208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077686.384212:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077686.384217:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961456128:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077686.384242:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961456256:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077686.384265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077686.384277:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077686/real 1769077686] req@ffff895e4383e300 x1855008961456256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077686.384291:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961456256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077686.384300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077686.384304:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077686.384307:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961456256:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077686.384314:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961456384:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077686.384334:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077686.384345:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077686/real 1769077686] req@ffff895e4383f800 x1855008961456384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077686.384357:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961456384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077686.384377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077686.384380:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077686.384384:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961456384:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077686.895831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077687.088011:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077687.919881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077688.111864:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077688.943823:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077689.135930:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077689.967856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077690.159902:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077690.991881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077691.183887:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077691.503839:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077691.503859:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077691.503864:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077691.503875:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3374 00000100:00080000:3.0:1769077691.503886:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077691.503893:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077691.503935:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961456512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077691.503959:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077691.503961:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077691.503963:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077691.503966:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3374 00000100:00080000:3.0:1769077691.503969:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077691.503972:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077691.503979:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077691.503988:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961456640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077691.503996:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077691.504014:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961456512:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077691.504016:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077691.504020:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077691.504027:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3374 00000100:00080000:3.0:1769077691.504031:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077691.504036:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077691.504043:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077691.504058:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961456768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077691.504082:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077691.504119:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077691/real 1769077691] req@ffff895e453d5500 x1855008961456512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077691.504165:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961456512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077691.504176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077691.504181:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077691.504185:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961456512:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077691.504193:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961456640:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077691.504216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077691.504230:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077691/real 1769077691] req@ffff895e453d7b80 x1855008961456640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077691.504243:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961456640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077691.504251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077691.504253:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077691.504256:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961456640:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077691.504274:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961456768:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077691.504290:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077691.504297:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077691/real 1769077691] req@ffff895e453d7100 x1855008961456768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077691.504305:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961456768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077691.504319:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077691.504322:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077691.504324:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961456768:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077692.015871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077692.207939:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077693.039864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077693.231920:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077694.063872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077694.255890:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077695.087855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077695.279903:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077696.111855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077696.303900:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077696.623868:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077696.623884:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077696.623889:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077696.623899:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3379 00000100:00080000:3.0:1769077696.623911:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077696.623918:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077696.623956:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961456896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077696.623979:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077696.623988:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077696.623990:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077696.623993:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3379 00000100:00080000:3.0:1769077696.623997:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077696.624000:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077696.624008:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077696.624032:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961456896:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077696.624036:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961457024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077696.624048:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077696.624050:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077696.624057:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077696.624060:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3379 00000100:00080000:3.0:1769077696.624064:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077696.624067:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077696.624071:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077696.624079:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961457152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077696.624109:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077696.624140:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077696/real 1769077696] req@ffff895e453d4700 x1855008961456896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077696.624154:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961456896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077696.624161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077696.624164:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077696.624167:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961456896:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077696.624186:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961457024:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077696.624201:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077696.624208:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077696/real 1769077696] req@ffff895e453d5880 x1855008961457024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077696.624216:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961457024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077696.624229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077696.624232:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077696.624234:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961457024:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077696.624238:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961457152:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077696.624254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077696.624264:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077696/real 1769077696] req@ffff895e453d5180 x1855008961457152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077696.624276:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961457152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077696.624285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077696.624289:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077696.624292:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961457152:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077697.135896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077697.327949:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077698.159873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077698.351907:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077699.183851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077699.375952:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077700.207847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077700.399887:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077701.231864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077701.423898:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077701.743886:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077701.743895:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077701.743898:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077701.743906:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3384 00000100:00080000:2.0:1769077701.743916:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077701.743923:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077701.743954:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961457280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077701.743983:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077701.743986:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077701.743987:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077701.743990:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3384 00000100:00080000:2.0:1769077701.743992:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077701.743994:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077701.744002:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077701.744026:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961457408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077701.744035:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077701.744038:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769077701.744041:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961457280:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077701.744041:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077701.744049:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3384 00000100:00080000:2.0:1769077701.744052:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077701.744055:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077701.744059:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077701.744085:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961457536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077701.744104:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077701.744139:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077701/real 1769077701] req@ffff895e44e2b480 x1855008961457280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077701.744154:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961457280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077701.744170:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077701.744174:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077701.744178:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961457280:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077701.744198:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961457408:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077701.744216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077701.744234:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077701/real 1769077701] req@ffff895e44e28e00 x1855008961457408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077701.744246:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961457408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077701.744252:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077701.744255:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077701.744257:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961457408:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077701.744261:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961457536:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077701.744273:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077701.744279:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077701/real 1769077701] req@ffff895e44e29f80 x1855008961457536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077701.744287:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961457536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077701.744292:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077701.744294:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077701.744296:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961457536:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077702.255851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077702.447862:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077703.279856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077703.471895:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077704.303870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077704.495992:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077705.327876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077705.519924:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077706.351878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077706.543872:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077706.863879:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077706.863889:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077706.863894:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077706.863903:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3389 00000100:00080000:2.0:1769077706.863913:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077706.863920:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077706.863956:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961457664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077706.863985:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077706.863989:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077706.863995:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077706.863999:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3389 00000100:00080000:2.0:1769077706.864018:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077706.864024:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077706.864034:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077706.864035:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961457664:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077706.864051:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961457792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077706.864064:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077706.864068:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077706.864070:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077706.864074:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3389 00000100:00080000:2.0:1769077706.864079:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077706.864083:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077706.864088:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077706.864100:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961457920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077706.864107:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077706.864154:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077706/real 1769077706] req@ffff895e44e29500 x1855008961457664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077706.864169:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961457664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077706.864177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077706.864180:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077706.864183:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961457664:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077706.864207:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961457792:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077706.864223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077706.864232:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077706/real 1769077706] req@ffff895e44e28a80 x1855008961457792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077706.864242:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961457792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077706.864251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077706.864255:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077706.864259:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961457792:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077706.864275:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961457920:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077706.864297:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077706.864309:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077706/real 1769077706] req@ffff895e44e29180 x1855008961457920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077706.864321:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961457920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077706.864329:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077706.864333:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077706.864336:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961457920:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077707.375854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077707.567880:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077708.400007:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077708.591854:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077709.423873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077709.615848:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077710.447843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077710.639922:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077711.471859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077711.663966:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077711.983914:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077711.983924:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077711.983929:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077711.983938:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3394 00000100:00080000:0.0:1769077711.983949:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077711.983956:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077711.983993:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961458048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077711.984019:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077711.984022:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077711.984040:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077711.984045:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3394 00000100:00080000:0.0:1769077711.984050:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077711.984054:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077711.984065:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077711.984075:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961458048:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077711.984082:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961458176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077711.984092:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077711.984094:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077711.984096:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077711.984099:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3394 00000100:00080000:0.0:1769077711.984101:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077711.984104:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077711.984108:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077711.984116:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961458304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077711.984159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077711.984214:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077711/real 1769077711] req@ffff895e4383ca80 x1855008961458048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077711.984234:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961458048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077711.984245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077711.984250:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077711.984262:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961458048:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077711.984287:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961458176:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077711.984307:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077711.984318:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077711/real 1769077711] req@ffff895e4383dc00 x1855008961458176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077711.984330:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961458176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077711.984338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077711.984342:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077711.984345:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961458176:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077711.984350:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961458304:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077711.984365:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077711.984373:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077711/real 1769077711] req@ffff895e4383ea00 x1855008961458304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077711.984383:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961458304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077711.984391:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077711.984394:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077711.984397:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961458304:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077712.495854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077712.687844:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077713.519878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077713.711894:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077714.543882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077714.735973:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077715.567884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077715.759891:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077716.591814:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077716.783947:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077717.103876:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077717.103890:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077717.103896:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077717.103910:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3399 00000100:00080000:2.0:1769077717.103923:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077717.103932:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077717.103972:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961458432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077717.104025:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077717.104029:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077717.104031:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077717.104035:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3399 00000100:00080000:2.0:1769077717.104039:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077717.104044:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077717.104055:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077717.104064:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961458432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077717.104070:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961458560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077717.104081:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077717.104085:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077717.104086:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077717.104090:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3399 00000100:00080000:2.0:1769077717.104094:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077717.104098:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077717.104102:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077717.104112:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961458688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077717.104156:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077717.104194:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077717/real 1769077717] req@ffff895e44e2bb80 x1855008961458432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077717.104209:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961458432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077717.104220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077717.104224:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077717.104229:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961458432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077717.104268:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961458560:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077717.104295:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077717.104308:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077717/real 1769077717] req@ffff895e44e2b800 x1855008961458560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077717.104320:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961458560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077717.104329:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077717.104332:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077717.104335:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961458560:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077717.104341:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961458688:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077717.104360:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077717.104371:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077717/real 1769077717] req@ffff895e44e29c00 x1855008961458688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077717.104398:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961458688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077717.104406:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077717.104421:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077717.104425:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961458688:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077717.615875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077717.807883:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077718.639898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077718.831888:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077719.663866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077719.855864:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077720.687880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077720.879899:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077721.711833:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077721.903811:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077722.223849:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077722.223874:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077722.223878:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077722.223887:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3404 00000100:00080000:0.0:1769077722.223897:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077722.223904:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077722.223945:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961458816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077722.223970:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077722.223973:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077722.223975:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077722.223979:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3404 00000100:00080000:0.0:1769077722.223983:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077722.223987:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077722.224000:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077722.224002:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961458816:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077722.224018:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961458944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077722.224030:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077722.224033:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077722.224035:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077722.224038:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3404 00000100:00080000:0.0:1769077722.224043:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077722.224047:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077722.224051:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077722.224054:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077722.224066:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961459072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077722.224085:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077722/real 1769077722] req@ffff895e4383c000 x1855008961458816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077722.224098:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961458816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077722.224105:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077722.224107:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077722.224110:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961458816:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077722.224126:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961458944:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077722.224138:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077722.224146:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077722/real 1769077722] req@ffff895e4383f480 x1855008961458944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077722.224152:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961458944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077722.224163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077722.224165:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077722.224167:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961458944:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077722.224170:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961459072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077722.224182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077722.224187:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077722/real 1769077722] req@ffff895e4383ce00 x1855008961459072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077722.224194:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961459072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077722.224198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077722.224200:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077722.224203:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961459072:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077722.735879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077722.927896:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077723.759878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077723.952068:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077724.783869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077724.975865:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077725.807819:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077725.999937:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077726.831879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077727.023872:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077727.343925:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077727.343936:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077727.343940:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077727.343949:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3409 00000100:00080000:3.0:1769077727.343961:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077727.343968:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077727.344002:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961459200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077727.344036:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077727.344053:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077727.344055:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077727.344059:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3409 00000100:00080000:3.0:1769077727.344063:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077727.344067:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077727.344075:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077727.344084:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961459200:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077727.344089:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961459328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077727.344104:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077727.344109:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077727.344110:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077727.344113:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3409 00000100:00080000:3.0:1769077727.344114:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077727.344117:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077727.344122:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077727.344134:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961459456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077727.344193:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077727.344270:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077727/real 1769077727] req@ffff895e453d6680 x1855008961459200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077727.344292:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961459200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077727.344303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077727.344308:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077727.344312:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961459200:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077727.344340:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961459328:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077727.344361:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077727.344372:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077727/real 1769077727] req@ffff895e453d4380 x1855008961459328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077727.344384:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961459328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077727.344392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077727.344395:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077727.344399:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961459328:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077727.344405:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961459456:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077727.344421:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077727.344441:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077727/real 1769077727] req@ffff895e453d6d80 x1855008961459456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077727.344452:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961459456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077727.344460:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077727.344463:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077727.344467:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961459456:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077727.855854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077728.047860:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077728.879840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077729.071914:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077729.903822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077730.095884:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077730.927898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077731.119869:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077731.951871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077732.143949:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077732.463836:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077732.463851:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077732.463856:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077732.463868:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3415 00000100:00080000:3.0:1769077732.463882:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077732.463912:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077732.463954:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961459584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077732.463983:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077732.463987:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077732.463989:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077732.463993:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3415 00000100:00080000:3.0:1769077732.463997:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077732.464001:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077732.464013:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077732.464029:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961459712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077732.464032:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961459584:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077732.464049:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077732.464052:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077732.464054:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077732.464059:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3415 00000100:00080000:3.0:1769077732.464063:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077732.464067:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077732.464072:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077732.464083:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961459840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077732.464173:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961459712:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077732.464200:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961459840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077732.975869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077733.167887:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077733.999894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077734.191937:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077735.023881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077735.215908:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769077735.536084:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769077735.536167:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077735.536193:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077735.536249:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077735.536260:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769077735.536865:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077732/real 1769077735] req@ffff895e453d6d80 x1855008961459712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077735.536910:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961459712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077735.536926:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077735.536953:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077735.536959:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961459712:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077735.536970:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077732/real 1769077735] req@ffff895e453d4380 x1855008961459840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077735.536986:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961459840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077735.536995:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077735.536999:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077735.537002:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961459840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077735.537011:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077732/real 1769077735] req@ffff895e453d6300 x1855008961459584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077735.537024:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961459584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077735.537032:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077735.537037:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077735.537048:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961459584:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077736.047860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077736.239963:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077737.071860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077737.263853:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077737.583887:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077737.583895:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077737.583898:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077737.583907:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3420 00000100:00080000:2.0:1769077737.583916:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077737.583922:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077737.583951:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961459968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077737.583981:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077737.583985:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077737.583986:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077737.583990:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3420 00000100:00080000:2.0:1769077737.583994:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077737.583999:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077737.584011:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077737.584025:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961460096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077737.584035:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077737.584038:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077737.584040:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077737.584059:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3420 00000100:00080000:2.0:1769077737.584063:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077737.584095:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077737.584102:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077737.584116:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961460224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077737.584127:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961459968:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769077737.584197:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077737.584317:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077737/real 1769077737] req@ffff895e44e29c00 x1855008961459968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077737.584353:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961459968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077737.584364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077737.584369:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077737.584374:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961459968:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077737.584382:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961460096:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077737.584409:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077737.584422:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077737/real 1769077737] req@ffff895e44e2b800 x1855008961460096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077737.584433:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961460096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077737.584441:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077737.584444:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077737.584448:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961460096:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077737.584480:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961460224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077737.584499:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077737.584510:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077737/real 1769077737] req@ffff895e44e2bb80 x1855008961460224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077737.584522:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961460224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077737.584544:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077737.584548:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077737.584551:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961460224:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077738.095870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077738.287967:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077739.119844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077739.311888:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077740.143849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077740.335875:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077741.167822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077741.359885:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077742.191932:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077742.383866:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077742.703866:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077742.703876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077742.703879:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077742.703888:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3425 00000100:00080000:3.0:1769077742.703897:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077742.703904:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077742.703938:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961460352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077742.703962:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077742.703965:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077742.703966:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077742.703968:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3425 00000100:00080000:3.0:1769077742.703971:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077742.703973:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077742.703993:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077742.704004:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961460480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077742.704011:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077742.704014:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077742.704015:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077742.704018:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3425 00000100:00080000:3.0:1769077742.704021:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077742.704023:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077742.704029:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961460352:192.168.204.117@tcp:250:kworker.0 00000008:00100000:3.0:1769077742.704043:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077742.704055:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961460608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077742.704095:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077742.704136:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077742/real 1769077742] req@ffff895e453d6680 x1855008961460352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077742.704157:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961460352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077742.704167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077742.704171:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077742.704173:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961460352:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077742.704179:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961460480:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077742.704198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077742.704207:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077742/real 1769077742] req@ffff895e453d5180 x1855008961460480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077742.704220:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961460480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077742.704242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077742.704246:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077742.704250:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961460480:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077742.704276:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961460608:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077742.704299:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077742.704311:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077742/real 1769077742] req@ffff895e453d5880 x1855008961460608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077742.704325:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961460608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077742.704333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077742.704336:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077742.704338:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961460608:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077743.215897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769077743.407953:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077744.239890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077744.431986:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077745.263862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077745.455995:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077746.287854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077746.479895:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077747.311835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077747.503938:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077747.823848:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077747.823857:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077747.823861:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077747.823869:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3430 00000100:00080000:3.0:1769077747.823881:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077747.823889:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077747.823956:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961460736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077747.823978:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077747.823986:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077747.823988:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077747.823995:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3430 00000100:00080000:3.0:1769077747.824000:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077747.824003:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077747.824011:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077747.824022:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961460736:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077747.824027:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961460864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077747.824042:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077747.824047:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077747.824066:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077747.824073:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3430 00000100:00080000:3.0:1769077747.824079:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077747.824085:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077747.824091:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077747.824106:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961460992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077747.824117:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077747.824177:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077747/real 1769077747] req@ffff895e453d4700 x1855008961460736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077747.824201:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961460736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077747.824214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077747.824219:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077747.824237:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961460736:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077747.824268:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961460864:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077747.824291:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077747.824301:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077747/real 1769077747] req@ffff895e453d7100 x1855008961460864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077747.824310:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961460864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077747.824317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077747.824319:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077747.824322:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961460864:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077747.824326:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961460992:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077747.824339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077747.824345:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077747/real 1769077747] req@ffff895e453d7b80 x1855008961460992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077747.824353:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961460992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077747.824358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077747.824360:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077747.824362:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961460992:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077748.335858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077748.527926:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077749.359893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077749.551996:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077750.383920:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077750.575977:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077751.407918:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077751.599923:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077752.431989:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077752.623919:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077752.943871:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077752.943880:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077752.943900:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077752.943908:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3435 00000100:00080000:0.0:1769077752.943918:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077752.943924:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077752.943955:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961461120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077752.943982:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077752.943985:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077752.943987:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077752.943992:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3435 00000100:00080000:0.0:1769077752.943996:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077752.944002:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077752.944012:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077752.944021:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961461120:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077752.944030:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961461248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077752.944043:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077752.944046:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077752.944048:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077752.944051:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3435 00000100:00080000:0.0:1769077752.944054:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077752.944057:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077752.944061:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077752.944070:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961461376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077752.944094:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077752.944136:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077752/real 1769077752] req@ffff895e4383c380 x1855008961461120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077752.944151:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961461120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077752.944159:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077752.944163:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077752.944166:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961461120:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077752.944184:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961461248:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077752.944202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077752.944210:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077752/real 1769077752] req@ffff895e4383f100 x1855008961461248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077752.944218:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961461248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077752.944236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077752.944240:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077752.944243:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961461248:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077752.944250:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961461376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077752.944272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077752.944284:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077752/real 1769077752] req@ffff895e4383fb80 x1855008961461376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077752.944299:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961461376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077752.944307:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077752.944311:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077752.944314:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961461376:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077753.455870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077753.647939:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077754.479921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077754.671957:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077755.503859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077755.695912:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077756.527892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077756.719952:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077757.551858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077757.743921:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077758.063868:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077758.063880:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077758.063885:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077758.063895:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3440 00000100:00080000:0.0:1769077758.063907:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077758.063915:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077758.063952:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961461504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077758.063978:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077758.063981:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077758.063982:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077758.063985:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3440 00000100:00080000:0.0:1769077758.063988:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077758.063991:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077758.063999:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077758.064008:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961461632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077758.064016:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077758.064019:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077758.064022:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769077758.064024:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961461504:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077758.064026:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3440 00000100:00080000:0.0:1769077758.064032:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077758.064034:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077758.064052:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077758.064062:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961461760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077758.064091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077758.064121:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077758/real 1769077758] req@ffff895e4383c700 x1855008961461504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077758.064135:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961461504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077758.064143:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077758.064153:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077758.064156:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961461504:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077758.064174:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961461632:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077758.064190:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077758.064199:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077758/real 1769077758] req@ffff895e4383e680 x1855008961461632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077758.064207:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961461632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077758.064213:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077758.064216:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077758.064218:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961461632:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077758.064222:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961461760:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077758.064232:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077758.064239:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077758/real 1769077758] req@ffff895e4383d880 x1855008961461760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077758.064246:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961461760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077758.064251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077758.064253:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077758.064255:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961461760:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077758.575839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077758.767898:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077759.599849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077759.791896:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077760.623843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077760.815890:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077761.647855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077761.839903:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077762.671853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077762.863899:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077763.183919:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077763.183931:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077763.183936:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077763.183944:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3445 00000100:00080000:0.0:1769077763.183955:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077763.183963:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077763.184003:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961461888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077763.184025:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077763.184028:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077763.184030:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077763.184034:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3445 00000100:00080000:0.0:1769077763.184037:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077763.184041:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077763.184051:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077763.184065:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961462016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077763.184078:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077763.184082:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961461888:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077763.184083:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077763.184088:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077763.184096:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3445 00000100:00080000:0.0:1769077763.184114:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077763.184117:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077763.184122:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077763.184133:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961462144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077763.184163:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077763.184200:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077763/real 1769077763] req@ffff895e4383d180 x1855008961461888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077763.184219:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961461888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077763.184229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077763.184233:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077763.184238:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961461888:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077763.184263:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961462016:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077763.184285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077763.184299:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077763/real 1769077763] req@ffff895e4383df80 x1855008961462016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077763.184312:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961462016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077763.184321:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077763.184325:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077763.184328:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961462016:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077763.184335:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961462144:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077763.184353:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077763.184364:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077763/real 1769077763] req@ffff895e4383ed80 x1855008961462144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077763.184393:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961462144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077763.184403:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077763.184406:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077763.184409:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961462144:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077763.695901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077763.887913:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077764.719839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077764.911980:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077765.743927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077765.935947:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077766.767877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077766.959948:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077767.791881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077767.983917:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077768.303903:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077768.303912:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077768.303916:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077768.303924:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3450 00000100:00080000:3.0:1769077768.303934:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077768.303942:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077768.303981:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961462272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077768.304018:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077768.304022:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077768.304024:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077768.304028:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3450 00000100:00080000:3.0:1769077768.304031:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077768.304035:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077768.304046:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077768.304060:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961462400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077768.304075:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961462272:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077768.304099:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077768.304104:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077768.304106:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077768.304112:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3450 00000100:00080000:3.0:1769077768.304118:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077768.304124:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077768.304131:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077768.304146:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961462528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077768.304164:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077768.304210:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077768/real 1769077768] req@ffff895e453d5500 x1855008961462272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077768.304227:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961462272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077768.304237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077768.304241:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077768.304244:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961462272:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077768.304269:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961462400:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077768.304286:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077768.304294:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077768/real 1769077768] req@ffff895e453d7480 x1855008961462400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077768.304303:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961462400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077768.304310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077768.304312:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077768.304333:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961462400:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077768.304338:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961462528:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077768.304361:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077768.304372:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077768/real 1769077768] req@ffff895e453d4e00 x1855008961462528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077768.304386:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961462528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077768.304395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077768.304398:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077768.304402:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961462528:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077768.815895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077769.007975:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077769.839868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077770.031913:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077770.863862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077771.055928:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077771.887860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077772.079900:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077772.911864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077773.103931:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077773.423837:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077773.423848:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077773.423852:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077773.423862:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3455 00000100:00080000:0.0:1769077773.423873:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077773.423881:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077773.423919:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961462656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077773.423948:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077773.423952:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077773.423955:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077773.423959:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3455 00000100:00080000:0.0:1769077773.423964:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769077773.423977:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961462656:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077773.423987:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077773.423999:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077773.424016:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961462784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077773.424028:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077773.424031:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077773.424033:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769077773.424038:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769077773.424040:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3455 00000100:00080000:0.0:1769077773.424047:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077773.424052:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077773.424058:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077773.424069:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961462912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077773.424078:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077773/real 1769077773] req@ffff895e4383ed80 x1855008961462656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077773.424093:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961462656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077773.424102:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077773.424105:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077773.424108:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961462656:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077773.424127:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961462784:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077773.424142:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077773.424150:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077773/real 1769077773] req@ffff895e4383df80 x1855008961462784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077773.424165:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961462784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077773.424171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077773.424173:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077773.424176:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961462784:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077773.424180:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961462912:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077773.424192:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077773.424199:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077773/real 1769077773] req@ffff895e4383d180 x1855008961462912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077773.424206:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961462912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077773.424211:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077773.424213:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077773.424214:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961462912:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077773.935871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077774.127896:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077774.959853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077775.151923:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077775.983862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077776.175919:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077777.007867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077777.199913:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077778.031857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077778.223890:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077778.543890:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077778.543911:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077778.543918:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077778.543931:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3461 00000100:00080000:1.0:1769077778.543944:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077778.543951:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077778.544003:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961463040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077778.544039:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077778.544042:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077778.544044:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077778.544047:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3461 00000100:00080000:1.0:1769077778.544067:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077778.544071:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077778.544081:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077778.544094:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961463168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077778.544104:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077778.544107:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077778.544109:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077778.544118:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3461 00000100:00100000:3.0:1769077778.544119:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961463040:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077778.544126:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077778.544137:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077778.544141:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077778.544153:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961463296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077778.544208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077778.544250:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077778/real 1769077778] req@ffff895e4713ce00 x1855008961463040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077778.544281:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961463040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077778.544314:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077778.544320:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077778.544325:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961463040:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077778.544352:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961463168:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077778.544410:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077778.544427:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077778/real 1769077778] req@ffff895e4713f480 x1855008961463168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077778.544440:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961463168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077778.544450:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077778.544454:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077778.544458:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961463168:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077778.544465:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961463296:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077778.544497:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077778.544510:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077778/real 1769077778] req@ffff895e4713ca80 x1855008961463296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077778.544523:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961463296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077778.544531:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077778.544534:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077778.544538:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961463296:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077779.055857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077779.247928:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077780.079885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077780.271901:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077781.103861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077781.295964:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077782.127865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077782.319921:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077783.151857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077783.343973:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077783.663878:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077783.663887:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077783.663890:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077783.663898:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3466 00000100:00080000:0.0:1769077783.663908:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077783.663914:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077783.663946:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961463424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077783.663972:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077783.663976:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077783.663994:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077783.663999:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3466 00000100:00080000:0.0:1769077783.664003:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077783.664007:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077783.664017:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077783.664027:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961463424:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077783.664031:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961463552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077783.664041:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077783.664044:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077783.664046:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077783.664050:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3466 00000100:00080000:0.0:1769077783.664053:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077783.664057:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077783.664060:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077783.664069:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961463680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077783.664085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077783.664123:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077783/real 1769077783] req@ffff895e4383d880 x1855008961463424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077783.664149:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961463424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077783.664156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077783.664159:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077783.664162:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961463424:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077783.664182:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961463552:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077783.664200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077783.664209:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077783/real 1769077783] req@ffff895e4383e680 x1855008961463552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077783.664216:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961463552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077783.664222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077783.664224:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077783.664227:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961463552:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077783.664230:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961463680:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077783.664240:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077783.664245:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077783/real 1769077783] req@ffff895e4383c700 x1855008961463680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077783.664260:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961463680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077783.664265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077783.664267:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077783.664269:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961463680:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077784.175813:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077784.367912:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077785.199853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077785.391893:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077786.223869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077786.416000:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077787.247873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077787.439945:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077788.271892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077788.463944:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077788.783903:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077788.783923:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077788.783932:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077788.783944:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3471 00000100:00080000:1.0:1769077788.783986:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077788.783994:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077788.784044:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961463808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077788.784075:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077788.784077:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077788.784079:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077788.784082:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3471 00000100:00080000:1.0:1769077788.784084:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077788.784087:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077788.784097:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077788.784106:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961463936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077788.784114:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077788.784117:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077788.784118:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077788.784121:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3471 00000100:00080000:1.0:1769077788.784123:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077788.784126:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077788.784130:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077788.784136:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961463808:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077788.784140:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961464064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077788.784246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077788.784319:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077788/real 1769077788] req@ffff895e4713f800 x1855008961463808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077788.784346:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961463808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077788.784360:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077788.784365:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077788.784371:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961463808:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077788.784380:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961463936:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077788.784408:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077788.784424:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077788/real 1769077788] req@ffff895e4713dc00 x1855008961463936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077788.784439:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961463936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077788.784468:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077788.784474:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077788.784479:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961463936:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077788.784513:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961464064:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077788.784540:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077788.784554:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077788/real 1769077788] req@ffff895e4713df80 x1855008961464064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077788.784568:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961464064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077788.784578:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077788.784601:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077788.784605:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961464064:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077789.295873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077789.487995:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077790.319885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077790.511995:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077791.343897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077791.535945:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077792.367926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077792.559991:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077793.391873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077793.583934:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077793.903888:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077793.903897:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077793.903901:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077793.903928:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3476 00000100:00080000:0.0:1769077793.903937:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077793.903943:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077793.903972:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961464192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077793.903997:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077793.903999:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077793.904006:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077793.904010:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3476 00000100:00080000:0.0:1769077793.904014:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077793.904018:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077793.904027:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077793.904045:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961464320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077793.904045:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961464192:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077793.904062:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077793.904066:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077793.904068:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077793.904071:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3476 00000100:00080000:0.0:1769077793.904074:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077793.904078:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077793.904082:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077793.904091:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961464448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077793.904108:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077793.904166:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077793/real 1769077793] req@ffff895e4383fb80 x1855008961464192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077793.904193:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961464192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077793.904201:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077793.904204:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077793.904215:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961464192:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077793.904234:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961464320:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077793.904252:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077793.904261:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077793/real 1769077793] req@ffff895e4383f100 x1855008961464320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077793.904278:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961464320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077793.904284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077793.904286:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077793.904289:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961464320:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077793.904293:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961464448:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077793.904305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077793.904312:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077793/real 1769077793] req@ffff895e4383c380 x1855008961464448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077793.904319:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961464448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077793.904324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077793.904326:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077793.904327:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961464448:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077794.415865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077794.607914:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077795.439829:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077795.631905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077796.463862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077796.655909:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077797.487941:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077797.680155:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077798.511896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077798.703949:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077799.023845:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077799.023868:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077799.023873:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077799.023883:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3481 00000100:00080000:0.0:1769077799.023895:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077799.023903:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077799.023941:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961464576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077799.023973:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077799.023976:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077799.023977:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077799.023981:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3481 00000100:00080000:0.0:1769077799.023984:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077799.023987:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077799.023994:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077799.024005:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961464704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077799.024013:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077799.024016:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077799.024017:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077799.024020:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3481 00000100:00080000:0.0:1769077799.024022:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077799.024027:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077799.024029:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961464576:192.168.204.117@tcp:250:kworker.0 00000008:00100000:0.0:1769077799.024057:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077799.024070:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961464832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077799.024135:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961464704:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077799.024181:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961464832:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077799.535831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077799.727867:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077800.559865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077800.751963:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077801.583951:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077801.775944:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769077802.096176:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769077802.096245:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077802.096300:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077802.096386:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077802.096415:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769077802.096559:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077799/real 1769077802] req@ffff895e4383c000 x1855008961464832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077802.096601:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961464832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077802.096621:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077802.096630:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077802.096637:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961464832:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077802.096652:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077799/real 1769077802] req@ffff895e4383ce00 x1855008961464576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077802.096675:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961464576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077802.096689:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077802.096696:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077802.096702:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961464576:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077802.096715:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077799/real 1769077802] req@ffff895e4383f480 x1855008961464704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077802.096738:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961464704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077802.097707:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077802.097716:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077802.097801:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961464704:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077802.607926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077802.800029:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077803.631824:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077803.823908:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077804.143892:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077804.143904:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077804.143910:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077804.143920:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3486 00000100:00080000:3.0:1769077804.143931:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077804.143939:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077804.143986:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961464960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077804.144017:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077804.144021:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077804.144023:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077804.144027:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3486 00000100:00080000:3.0:1769077804.144031:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077804.144035:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077804.144045:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077804.144060:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961465088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077804.144073:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077804.144074:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961464960:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077804.144081:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077804.144085:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077804.144090:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3486 00000100:00080000:3.0:1769077804.144095:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077804.144116:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077804.144122:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077804.144135:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961465216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077804.144145:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077804.144187:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077804/real 1769077804] req@ffff895e453d4a80 x1855008961464960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077804.144207:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961464960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077804.144216:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077804.144220:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077804.144224:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961464960:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077804.144250:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961465088:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077804.144275:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077804.144289:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077804/real 1769077804] req@ffff895e453d5c00 x1855008961465088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077804.144301:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961465088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077804.144322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077804.144326:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077804.144329:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961465088:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077804.144336:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961465216:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077804.144358:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077804.144369:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077804/real 1769077804] req@ffff895e453d7800 x1855008961465216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077804.144379:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961465216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077804.144387:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077804.144390:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077804.144393:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961465216:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077804.655846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077804.847878:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077805.679840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077805.871915:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077806.703874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077806.895882:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077807.727836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077807.919912:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077808.751843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077808.943929:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077809.263911:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077809.263923:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077809.263928:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077809.263938:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3491 00000100:00080000:0.0:1769077809.263950:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077809.263958:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077809.264025:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961465344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077809.264058:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077809.264062:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077809.264064:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077809.264069:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3491 00000100:00080000:0.0:1769077809.264074:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077809.264079:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077809.264090:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077809.264090:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961465344:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077809.264110:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961465472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077809.264122:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077809.264126:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077809.264129:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077809.264132:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3491 00000100:00080000:0.0:1769077809.264137:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077809.264141:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077809.264147:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077809.264147:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077809.264163:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961465600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077809.264188:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077809/real 1769077809] req@ffff895e4383ea00 x1855008961465344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077809.264206:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961465344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077809.264217:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077809.264222:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077809.264226:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961465344:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077809.264250:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961465472:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077809.264273:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077809.264287:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077809/real 1769077809] req@ffff895e4383dc00 x1855008961465472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077809.264301:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961465472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077809.264309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077809.264312:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077809.264314:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961465472:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077809.264325:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961465600:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077809.264343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077809.264351:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077809/real 1769077809] req@ffff895e4383ca80 x1855008961465600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077809.264359:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961465600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077809.264364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077809.264366:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077809.264369:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961465600:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077809.775881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077809.967964:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077810.799870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077810.991949:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077811.823914:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077812.015977:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077812.847901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077813.039919:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077813.871873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077814.063970:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077814.383863:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077814.383876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077814.383898:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077814.383908:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3496 00000100:00080000:0.0:1769077814.383919:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077814.383928:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077814.383965:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961465728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077814.383992:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077814.383995:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077814.383997:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077814.384001:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3496 00000100:00080000:0.0:1769077814.384005:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077814.384008:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077814.384016:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077814.384022:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961465728:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077814.384033:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961465856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077814.384046:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077814.384050:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077814.384052:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077814.384057:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3496 00000100:00080000:0.0:1769077814.384062:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077814.384066:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077814.384072:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077814.384085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077814.384086:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961465984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077814.384120:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077814/real 1769077814] req@ffff895e4383f800 x1855008961465728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077814.384137:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961465728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077814.384151:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077814.384155:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077814.384159:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961465728:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077814.384181:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961465856:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077814.384201:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077814.384213:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077814/real 1769077814] req@ffff895e4383e300 x1855008961465856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077814.384230:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961465856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077814.384245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077814.384247:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077814.384250:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961465856:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077814.384254:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961465984:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077814.384268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077814.384276:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077814/real 1769077814] req@ffff895e4383d500 x1855008961465984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077814.384284:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961465984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077814.384289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077814.384291:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077814.384293:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961465984:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077814.895868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077815.087913:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077815.919856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077816.111953:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077816.943895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077817.135929:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077817.967887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077818.159999:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077818.991841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077819.183924:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077819.503843:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077819.503852:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077819.503856:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077819.503864:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3502 00000100:00080000:0.0:1769077819.503874:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077819.503888:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077819.503918:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961466112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077819.503942:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077819.503945:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077819.503946:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077819.503949:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3502 00000100:00080000:0.0:1769077819.503951:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077819.503954:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077819.503960:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077819.503971:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961466240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077819.503979:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077819.503982:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077819.503983:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077819.503986:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3502 00000100:00100000:3.0:1769077819.503991:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961466112:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077819.504007:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077819.504012:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077819.504016:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077819.504026:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961466368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077819.504046:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077819.504077:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077819/real 1769077819] req@ffff895e4383d500 x1855008961466112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077819.504090:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961466112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077819.504108:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077819.504111:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077819.504114:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961466112:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077819.504120:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961466240:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077819.504144:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077819.504157:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077819/real 1769077819] req@ffff895e4383e300 x1855008961466240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077819.504168:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961466240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077819.504177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077819.504181:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077819.504184:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961466240:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077819.504206:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961466368:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077819.504226:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077819.504238:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077819/real 1769077819] req@ffff895e4383f800 x1855008961466368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077819.504251:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961466368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077819.504266:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077819.504270:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077819.504274:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961466368:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077820.015858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077820.207898:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077821.039820:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077821.231898:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077822.063858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077822.255902:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077823.087851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077823.279958:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077824.111859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077824.303933:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077824.623848:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077824.623858:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077824.623863:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077824.623873:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3507 00000100:00080000:1.0:1769077824.623883:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077824.623890:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077824.623927:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961466496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077824.623957:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077824.623961:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077824.623963:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077824.623967:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3507 00000100:00080000:1.0:1769077824.623971:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077824.623975:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077824.623986:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077824.624004:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961466624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077824.624004:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961466496:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077824.624022:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077824.624026:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077824.624028:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077824.624033:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3507 00000100:00080000:1.0:1769077824.624038:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077824.624056:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077824.624062:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769077824.624070:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769077824.624079:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961466752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077824.624111:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077824/real 1769077824] req@ffff895e4713d180 x1855008961466496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077824.624131:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961466496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077824.624142:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077824.624146:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077824.624151:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961466496:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077824.624173:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961466624:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077824.624193:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077824.624205:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077824/real 1769077824] req@ffff895e4713c380 x1855008961466624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077824.624234:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961466624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077824.624243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077824.624247:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077824.624251:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961466624:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077824.624257:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961466752:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077824.624275:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077824.624285:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077824/real 1769077824] req@ffff895e4713df80 x1855008961466752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077824.624305:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961466752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077824.624313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077824.624317:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077824.624320:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961466752:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077825.135865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077825.327910:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077826.159836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077826.351918:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077827.183842:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077827.375896:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077828.207847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077828.399868:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077829.231849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077829.423913:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077829.743824:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077829.743833:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077829.743837:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077829.743845:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3512 00000100:00080000:1.0:1769077829.743855:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077829.743860:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077829.743891:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961466880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077829.743914:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077829.743916:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077829.743918:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077829.743920:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3512 00000100:00080000:1.0:1769077829.743922:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077829.743925:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077829.743932:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077829.743941:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961467008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077829.743950:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769077829.743955:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961466880:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077829.743968:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077829.743970:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077829.743974:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3512 00000100:00080000:1.0:1769077829.743978:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077829.743981:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077829.743985:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077829.743995:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961467136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077829.744013:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077829.744046:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077829/real 1769077829] req@ffff895e4713dc00 x1855008961466880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077829.744060:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961466880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077829.744068:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077829.744071:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077829.744074:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961466880:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077829.744094:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961467008:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077829.744114:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077829.744126:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077829/real 1769077829] req@ffff895e4713f800 x1855008961467008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077829.744140:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961467008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077829.744163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077829.744166:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077829.744168:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961467008:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077829.744173:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961467136:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077829.744193:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077829.744204:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077829/real 1769077829] req@ffff895e4713ca80 x1855008961467136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077829.744217:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961467136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077829.744226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077829.744230:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077829.744233:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961467136:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077830.255939:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077830.447910:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077831.279892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077831.471961:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077832.303893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077832.495934:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077833.327889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077833.519925:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077834.351848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077834.543907:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077834.863869:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077834.863881:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077834.863886:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077834.863896:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3517 00000100:00080000:3.0:1769077834.863908:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077834.863915:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077834.863956:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961467264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077834.863994:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077834.864015:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077834.864017:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077834.864024:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3517 00000100:00080000:3.0:1769077834.864029:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077834.864035:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077834.864046:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077834.864061:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961467264:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077834.864062:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961467392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077834.864085:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077834.864088:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077834.864091:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077834.864095:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3517 00000100:00080000:3.0:1769077834.864099:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077834.864103:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077834.864107:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077834.864117:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961467520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077834.864150:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077834.864186:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077834/real 1769077834] req@ffff895e453d5f80 x1855008961467264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077834.864203:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961467264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077834.864215:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077834.864220:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077834.864224:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961467264:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077834.864264:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961467392:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077834.864287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077834.864297:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077834/real 1769077834] req@ffff895e453d7800 x1855008961467392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077834.864306:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961467392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077834.864330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077834.864335:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077834.864339:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961467392:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077834.864356:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961467520:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077834.864384:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077834.864394:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077834/real 1769077834] req@ffff895e453d5c00 x1855008961467520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077834.864403:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961467520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077834.864408:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077834.864411:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077834.864413:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961467520:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077835.375927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077835.567956:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077836.399830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077836.591880:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077837.423932:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077837.615919:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077838.447811:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077838.639898:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077839.471844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077839.663879:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077839.983873:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077839.983883:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077839.983887:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077839.983894:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3522 00000100:00080000:1.0:1769077839.983904:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077839.983911:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077839.983941:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961467648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077839.983964:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077839.983966:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077839.983980:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077839.983983:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3522 00000100:00080000:1.0:1769077839.983987:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077839.983990:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077839.983998:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077839.984008:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961467776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077839.984015:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077839.984018:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077839.984019:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077839.984023:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3522 00000100:00100000:0.0:1769077839.984026:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961467648:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077839.984026:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077839.984032:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077839.984038:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077839.984046:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961467904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077839.984100:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077839.984149:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077839/real 1769077839] req@ffff895e4713f480 x1855008961467648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077839.984167:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961467648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077839.984178:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077839.984182:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077839.984194:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961467648:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077839.984218:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961467776:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077839.984243:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077839.984257:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077839/real 1769077839] req@ffff895e4713ce00 x1855008961467776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077839.984270:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961467776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077839.984278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077839.984281:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077839.984283:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961467776:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077839.984288:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961467904:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077839.984303:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077839.984311:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077839/real 1769077839] req@ffff895e4713d500 x1855008961467904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077839.984318:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961467904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077839.984331:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077839.984334:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077839.984336:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961467904:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077840.495821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077840.687922:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077841.519888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077841.711962:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077842.543816:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077842.735918:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077843.567918:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077843.759923:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077844.591855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077844.783913:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077845.103917:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077845.103930:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077845.103947:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077845.103960:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3527 00000100:00080000:1.0:1769077845.103974:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077845.103983:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077845.104071:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961468032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077845.104106:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077845.104110:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077845.104112:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077845.104117:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3527 00000100:00080000:1.0:1769077845.104121:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077845.104126:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077845.104137:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077845.104150:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961468160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077845.104162:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077845.104166:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077845.104168:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077845.104174:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961468032:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077845.104175:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3527 00000100:00080000:1.0:1769077845.104181:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077845.104188:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077845.104194:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077845.104208:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961468288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077845.104291:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077845.104342:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077845/real 1769077845] req@ffff895e4713ea00 x1855008961468032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077845.104359:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961468032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077845.104367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077845.104371:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077845.104374:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961468032:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077845.104399:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961468160:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077845.104417:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077845.104426:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077845/real 1769077845] req@ffff895e4713f100 x1855008961468160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077845.104434:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961468160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077845.104441:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077845.104443:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077845.104445:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961468160:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077845.104449:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961468288:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077845.104459:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077845.104464:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077845/real 1769077845] req@ffff895e4713e300 x1855008961468288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077845.104472:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961468288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077845.104487:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077845.104489:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077845.104492:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961468288:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077845.615892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077845.807903:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077846.639869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077846.831954:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077847.663869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077847.855887:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077848.687855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077848.879921:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077849.711853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077849.903898:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077850.223871:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077850.223880:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077850.223895:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077850.223903:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3532 00000100:00080000:0.0:1769077850.223921:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077850.223927:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077850.223971:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961468416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077850.223994:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077850.223996:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077850.223998:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077850.224000:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3532 00000100:00080000:0.0:1769077850.224003:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077850.224005:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077850.224012:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077850.224021:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961468544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077850.224029:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077850.224032:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769077850.224040:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961468416:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077850.224048:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077850.224053:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3532 00000100:00080000:0.0:1769077850.224056:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077850.224060:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077850.224064:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077850.224074:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961468672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077850.224100:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077850.224133:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077850/real 1769077850] req@ffff895e4383ca80 x1855008961468416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077850.224146:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961468416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077850.224171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077850.224202:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077850.224207:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961468416:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077850.224216:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961468544:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077850.224242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077850.224252:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077850/real 1769077850] req@ffff895e4383dc00 x1855008961468544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077850.224261:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961468544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077850.224268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077850.224270:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077850.224272:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961468544:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077850.224291:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961468672:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077850.224303:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077850.224309:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077850/real 1769077850] req@ffff895e4383ea00 x1855008961468672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077850.224317:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961468672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077850.224322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077850.224324:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077850.224326:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961468672:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077850.735880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077850.927951:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077851.759823:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077851.951868:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077852.783929:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077852.975974:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077853.807882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077853.999884:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077854.831858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077855.023964:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077855.343864:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077855.343874:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077855.343877:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077855.343885:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3537 00000100:00080000:0.0:1769077855.343895:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077855.343901:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077855.343952:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961468800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077855.343983:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077855.343987:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077855.343989:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077855.343993:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3537 00000100:00080000:0.0:1769077855.343997:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077855.344002:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077855.344012:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077855.344026:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961468928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077855.344055:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961468800:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077855.344072:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077855.344075:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077855.344076:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077855.344080:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3537 00000100:00080000:0.0:1769077855.344084:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077855.344103:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077855.344108:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077855.344135:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961469056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077855.344160:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077855.344193:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077855/real 1769077855] req@ffff895e4383f480 x1855008961468800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077855.344224:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961468800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077855.344233:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077855.344236:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077855.344239:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961468800:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077855.344257:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961468928:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077855.344274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077855.344282:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077855/real 1769077855] req@ffff895e4383ce00 x1855008961468928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077855.344290:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961468928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077855.344297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077855.344300:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077855.344302:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961468928:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077855.344306:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961469056:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077855.344316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077855.344322:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077855/real 1769077855] req@ffff895e4383c000 x1855008961469056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077855.344337:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961469056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077855.344359:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077855.344361:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077855.344363:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961469056:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077855.855833:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077856.047923:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077856.879817:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077857.071897:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077857.903908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077858.095958:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077858.927855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077859.119927:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077859.951845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077860.143927:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077860.463825:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077860.463834:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077860.463838:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077860.463848:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3543 00000100:00080000:0.0:1769077860.463871:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077860.463879:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077860.463916:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961469184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077860.463938:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077860.463941:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077860.463942:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077860.463945:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3543 00000100:00080000:0.0:1769077860.463958:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077860.463961:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077860.463984:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077860.463997:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961469312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077860.463997:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961469184:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769077860.464022:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077860.464025:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077860.464026:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077860.464034:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3543 00000100:00080000:0.0:1769077860.464038:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077860.464051:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077860.464055:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077860.464066:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961469440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077860.464102:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077860.464133:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077860/real 1769077860] req@ffff895e4383c380 x1855008961469184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077860.464147:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961469184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077860.464155:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077860.464158:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077860.464161:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961469184:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077860.464179:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961469312:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077860.464193:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077860.464201:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077860/real 1769077860] req@ffff895e4383f100 x1855008961469312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077860.464209:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961469312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077860.464225:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077860.464228:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077860.464230:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961469312:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077860.464235:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961469440:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077860.464249:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077860.464255:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077860/real 1769077860] req@ffff895e4383fb80 x1855008961469440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077860.464262:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961469440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077860.464267:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077860.464269:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077860.464271:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961469440:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077860.975866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077861.167948:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077861.999857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077862.191961:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077863.023886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077863.215939:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077864.047894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077864.239980:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077865.071900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077865.264006:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077865.583941:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077865.583957:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077865.583963:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077865.583977:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3548 00000100:00080000:3.0:1769077865.583992:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077865.584001:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077865.584055:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961469568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077865.584095:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077865.584117:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077865.584119:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077865.584124:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3548 00000100:00080000:3.0:1769077865.584128:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077865.584132:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077865.584145:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077865.584156:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961469568:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077865.584167:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961469696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077865.584179:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077865.584182:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077865.584184:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077865.584188:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3548 00000100:00080000:3.0:1769077865.584192:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077865.584195:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077865.584199:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077865.584215:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961469824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077865.584295:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961469696:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077865.584336:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961469824:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077866.095903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077866.287964:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077867.119901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077867.311897:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077868.143904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077868.335970:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769077868.656098:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769077868.656156:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769077868.656186:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769077868.656244:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769077868.656254:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769077868.656322:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077865/real 1769077868] req@ffff895e453d4e00 x1855008961469696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077868.656364:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961469696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077868.656377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077868.656384:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077868.656421:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961469696:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077868.656432:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077865/real 1769077868] req@ffff895e453d7480 x1855008961469824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077868.656448:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961469824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077868.656456:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077868.656460:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077868.656464:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961469824:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077868.656472:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077865/real 1769077868] req@ffff895e453d4a80 x1855008961469568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077868.656485:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961469568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077868.656493:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077868.656497:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077868.656500:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961469568:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077869.167902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077869.359962:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077870.191897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077870.383892:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077870.703882:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077870.703893:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077870.703897:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077870.703905:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3553 00000100:00080000:1.0:1769077870.703915:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077870.703922:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077870.703955:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961469952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077870.703980:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077870.703983:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077870.703985:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077870.703988:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3553 00000100:00080000:1.0:1769077870.703990:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077870.703992:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077870.704000:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077870.704027:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961470080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077870.704033:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961469952:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077870.704044:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077870.704048:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077870.704050:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077870.704053:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3553 00000100:00080000:1.0:1769077870.704059:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077870.704064:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077870.704067:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077870.704082:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961470208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077870.704132:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077870.704191:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077870/real 1769077870] req@ffff895e4713d880 x1855008961469952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077870.704229:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961469952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077870.704255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077870.704262:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077870.704267:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961469952:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077870.704310:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961470080:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077870.704342:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077870.704356:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077870/real 1769077870] req@ffff895e4713ed80 x1855008961470080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077870.704369:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961470080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077870.704381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077870.704401:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077870.704405:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961470080:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077870.704413:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961470208:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077870.704435:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077870.704448:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077870/real 1769077870] req@ffff895e4713c000 x1855008961470208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077870.704460:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961470208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077870.704469:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077870.704473:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077870.704476:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961470208:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077871.215924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769077871.407907:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077872.239874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077872.431917:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077873.263857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769077873.455877:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077874.287913:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077874.479975:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077875.311848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077875.503876:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077875.823911:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077875.823924:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077875.823928:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077875.823938:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3558 00000100:00080000:2.0:1769077875.823949:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077875.823956:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077875.824005:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961470336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077875.824035:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077875.824039:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077875.824048:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077875.824067:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3558 00000100:00080000:2.0:1769077875.824072:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077875.824077:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077875.824086:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077875.824087:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961470336:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077875.824103:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961470464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077875.824116:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077875.824120:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077875.824123:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077875.824127:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3558 00000100:00080000:2.0:1769077875.824131:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077875.824135:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077875.824141:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077875.824150:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769077875.824154:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961470592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077875.824194:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077875/real 1769077875] req@ffff895e44e29180 x1855008961470336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077875.824216:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961470336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077875.824227:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077875.824232:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077875.824236:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961470336:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077875.824260:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961470464:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077875.824280:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077875.824293:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077875/real 1769077875] req@ffff895e44e28a80 x1855008961470464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077875.824307:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961470464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077875.824316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077875.824319:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077875.824323:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961470464:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077875.824330:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961470592:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077875.824350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077875.824365:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077875/real 1769077875] req@ffff895e44e29500 x1855008961470592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077875.824374:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961470592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077875.824379:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077875.824381:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077875.824383:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961470592:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077876.335865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077876.527933:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077877.359807:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077877.551964:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077878.383907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077878.575899:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077879.407914:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077879.599954:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077880.431845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077880.623912:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077880.943880:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077880.943891:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077880.943897:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077880.943914:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3563 00000100:00080000:2.0:1769077880.943924:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077880.943931:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077880.943974:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961470720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077880.944017:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077880.944020:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077880.944023:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077880.944027:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3563 00000100:00080000:2.0:1769077880.944032:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077880.944037:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077880.944046:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961470720:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769077880.944047:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077880.944064:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961470848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077880.944075:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077880.944078:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077880.944080:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077880.944084:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3563 00000100:00080000:2.0:1769077880.944088:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077880.944092:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077880.944105:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077880.944110:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769077880.944121:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961470976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077880.944151:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077880/real 1769077880] req@ffff895e44e29f80 x1855008961470720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077880.944170:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961470720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077880.944180:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077880.944185:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077880.944189:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961470720:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077880.944213:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961470848:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077880.944236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077880.944248:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077880/real 1769077880] req@ffff895e44e28e00 x1855008961470848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077880.944260:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961470848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077880.944281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077880.944285:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077880.944288:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961470848:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077880.944294:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961470976:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077880.944315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077880.944326:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077880/real 1769077880] req@ffff895e44e2b480 x1855008961470976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077880.944337:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961470976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077880.944345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077880.944348:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077880.944351:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961470976:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077881.455849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077881.647872:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077882.479852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077882.672037:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077883.503889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077883.695986:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077884.528012:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077884.719960:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077885.551875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077885.743950:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077886.063886:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077886.063899:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077886.063905:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077886.063917:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3568 00000100:00080000:0.0:1769077886.063930:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077886.063937:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077886.063981:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961471104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077886.064022:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077886.064043:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077886.064045:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077886.064050:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3568 00000100:00080000:0.0:1769077886.064054:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077886.064059:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077886.064076:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961471104:192.168.204.117@tcp:250:kworker.0 00000008:00100000:0.0:1769077886.064080:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077886.064097:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961471232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077886.064108:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077886.064112:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077886.064114:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077886.064119:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3568 00000100:00080000:0.0:1769077886.064122:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077886.064126:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077886.064131:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077886.064140:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961471360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077886.064152:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077886.064204:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077886/real 1769077886] req@ffff895e4383c700 x1855008961471104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077886.064224:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961471104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077886.064245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077886.064250:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077886.064254:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961471104:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077886.064282:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961471232:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077886.064303:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077886.064312:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077886/real 1769077886] req@ffff895e4383e680 x1855008961471232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077886.064320:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961471232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077886.064327:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077886.064330:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077886.064333:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961471232:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077886.064339:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961471360:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077886.064356:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077886.064366:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077886/real 1769077886] req@ffff895e4383d880 x1855008961471360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077886.064399:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961471360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077886.064407:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077886.064410:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077886.064413:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961471360:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077886.575854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077886.767916:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077887.599868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077887.791897:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077888.623873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077888.815883:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077889.647805:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077889.839871:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077890.671851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077890.863940:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077891.183866:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077891.183876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077891.183879:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077891.183887:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3573 00000100:00080000:2.0:1769077891.183914:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077891.183920:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077891.183949:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961471488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077891.183971:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077891.183973:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077891.183975:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077891.183978:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3573 00000100:00080000:2.0:1769077891.183981:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077891.183984:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077891.183994:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077891.184008:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961471616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077891.184019:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077891.184022:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077891.184024:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077891.184031:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3573 00000100:00080000:2.0:1769077891.184036:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077891.184046:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077891.184050:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961471488:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769077891.184052:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077891.184068:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961471744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077891.184149:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077891.184194:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077891/real 1769077891] req@ffff895e44e2aa00 x1855008961471488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077891.184227:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961471488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077891.184239:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077891.184243:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077891.184246:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961471488:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077891.184252:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961471616:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077891.184271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077891.184280:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077891/real 1769077891] req@ffff895e44e2b100 x1855008961471616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077891.184305:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961471616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077891.184314:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077891.184318:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077891.184321:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961471616:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077891.184354:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961471744:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077891.184370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077891.184378:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077891/real 1769077891] req@ffff895e44e2a300 x1855008961471744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077891.184385:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961471744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077891.184390:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077891.184393:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077891.184394:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961471744:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077891.695866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077891.887948:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077892.719845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077892.911955:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077893.743852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077893.935930:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077894.768076:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077894.959893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077895.791852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077895.983907:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077896.303881:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077896.303893:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077896.303897:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077896.303908:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3578 00000100:00080000:3.0:1769077896.303919:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077896.303943:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077896.303972:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961471872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077896.303998:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077896.304001:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077896.304002:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077896.304005:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3578 00000100:00080000:3.0:1769077896.304009:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077896.304011:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077896.304019:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077896.304028:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961472000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077896.304036:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077896.304038:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077896.304039:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077896.304042:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3578 00000100:00080000:3.0:1769077896.304044:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077896.304049:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077896.304052:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961471872:192.168.204.117@tcp:250:kworker.0 00000008:00100000:3.0:1769077896.304053:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077896.304062:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961472128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077896.304119:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077896.304157:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077896/real 1769077896] req@ffff895e453d5500 x1855008961471872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077896.304176:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961471872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077896.304201:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077896.304206:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077896.304211:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961471872:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077896.304219:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961472000:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077896.304243:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077896.304257:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077896/real 1769077896] req@ffff895e453d7b80 x1855008961472000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077896.304271:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961472000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077896.304280:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077896.304284:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077896.304288:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961472000:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077896.304311:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961472128:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077896.304331:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077896.304355:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077896/real 1769077896] req@ffff895e453d7100 x1855008961472128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077896.304369:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961472128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077896.304378:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077896.304382:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077896.304385:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961472128:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769077896.815837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077897.007900:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769077897.839869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077898.031951:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077898.863870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077899.055959:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077899.887944:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077900.079946:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077900.911820:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077901.103933:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077901.423823:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077901.423849:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077901.423852:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077901.423860:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3583 00000100:00080000:3.0:1769077901.423870:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077901.423875:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077901.423903:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961472256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077901.423925:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077901.423928:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077901.423929:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077901.423932:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3583 00000100:00080000:3.0:1769077901.423935:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077901.423938:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077901.423948:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077901.423963:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961472384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077901.423973:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961472256:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077901.423980:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077901.423985:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077901.423988:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077901.423993:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3583 00000100:00080000:3.0:1769077901.423998:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077901.424003:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077901.424008:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077901.424018:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961472512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077901.424036:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077901.424071:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077901/real 1769077901] req@ffff895e453d4700 x1855008961472256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077901.424084:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961472256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077901.424092:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077901.424095:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077901.424098:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961472256:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077901.424115:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961472384:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077901.424133:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077901.424144:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077901/real 1769077901] req@ffff895e453d5880 x1855008961472384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077901.424157:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961472384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077901.424165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077901.424169:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077901.424172:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961472384:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077901.424178:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961472512:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077901.424214:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077901.424227:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077901/real 1769077901] req@ffff895e453d5180 x1855008961472512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077901.424250:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961472512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077901.424255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077901.424257:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077901.424259:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961472512:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077901.935849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077902.127928:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077902.959852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077903.151936:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077903.983850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077904.175936:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077905.007844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077905.199934:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077906.031825:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077906.223885:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077906.543911:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077906.543919:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077906.543923:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077906.543938:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3589 00000100:00080000:2.0:1769077906.543949:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077906.543955:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077906.543987:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961472640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077906.544017:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077906.544021:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077906.544023:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077906.544027:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3589 00000100:00080000:2.0:1769077906.544030:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077906.544035:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077906.544045:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077906.544057:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961472768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077906.544068:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077906.544072:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077906.544074:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077906.544078:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3589 00000100:00080000:2.0:1769077906.544083:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769077906.544100:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961472640:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077906.544106:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077906.544113:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077906.544125:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961472896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077906.544159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077906.544192:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077906/real 1769077906] req@ffff895e44e28000 x1855008961472640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077906.544206:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961472640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077906.544213:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077906.544216:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077906.544219:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961472640:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077906.544224:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961472768:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077906.544265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077906.544274:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077906/real 1769077906] req@ffff895e44e28380 x1855008961472768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077906.544283:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961472768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077906.544288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077906.544290:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077906.544292:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961472768:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077906.544312:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961472896:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077906.544322:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077906.544328:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077906/real 1769077906] req@ffff895e44e29880 x1855008961472896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077906.544335:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961472896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077906.544340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077906.544341:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077906.544343:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961472896:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077907.055858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077907.247905:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077908.079865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077908.271864:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077909.103876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077909.295930:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077910.127849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077910.319880:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077911.151881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077911.343912:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077911.663889:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077911.663898:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077911.663902:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077911.663910:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3594 00000100:00080000:1.0:1769077911.663920:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077911.663925:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077911.663957:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961473024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077911.664016:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077911.664019:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077911.664021:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077911.664024:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3594 00000100:00080000:1.0:1769077911.664027:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077911.664029:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077911.664053:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077911.664064:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961473152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077911.664072:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077911.664075:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077911.664077:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077911.664080:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3594 00000100:00080000:1.0:1769077911.664082:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077911.664088:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077911.664091:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961473024:192.168.204.117@tcp:250:kworker.0 00000008:00100000:1.0:1769077911.664092:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077911.664101:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961473280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077911.664213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077911.664253:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077911/real 1769077911] req@ffff895e4713fb80 x1855008961473024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077911.664273:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961473024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077911.664293:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077911.664298:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077911.664302:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961473024:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077911.664308:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961473152:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077911.664327:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077911.664337:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077911/real 1769077911] req@ffff895e4713c700 x1855008961473152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077911.664345:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961473152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077911.664351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077911.664353:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077911.664355:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961473152:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077911.664381:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961473280:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077911.664400:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077911.664411:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077911/real 1769077911] req@ffff895e4713c000 x1855008961473280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077911.664439:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961473280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077911.664450:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077911.664454:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077911.664458:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961473280:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077912.175848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077912.367855:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077913.199877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077913.391917:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077914.223846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077914.415928:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077915.247856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077915.439916:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077916.271837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077916.463939:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077916.783909:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077916.783920:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077916.783925:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077916.783936:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3599 00000100:00080000:2.0:1769077916.783947:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077916.783954:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077916.783994:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961473408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077916.784024:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077916.784027:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077916.784030:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077916.784034:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3599 00000100:00080000:2.0:1769077916.784038:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077916.784042:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077916.784052:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077916.784088:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961473408:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077916.784101:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961473536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077916.784114:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077916.784118:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077916.784136:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077916.784140:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3599 00000100:00080000:2.0:1769077916.784146:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769077916.784147:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:2.0:1769077916.784155:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077916.784162:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077916.784177:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961473664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077916.784186:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077916/real 1769077916] req@ffff895e44e29880 x1855008961473408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077916.784201:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961473408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077916.784209:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077916.784212:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077916.784215:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961473408:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077916.784235:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961473536:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077916.784250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077916.784259:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077916/real 1769077916] req@ffff895e44e28380 x1855008961473536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077916.784275:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961473536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077916.784281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077916.784283:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077916.784285:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961473536:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077916.784289:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961473664:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077916.784302:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077916.784308:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077916/real 1769077916] req@ffff895e44e28000 x1855008961473664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077916.784316:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961473664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077916.784321:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077916.784323:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077916.784325:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961473664:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077917.295872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077917.487921:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077918.319818:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077918.511929:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077919.343859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077919.535925:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077920.367852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077920.559934:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077921.391873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077921.583956:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077921.903943:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077921.903958:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077921.903965:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077921.904006:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3604 00000100:00080000:1.0:1769077921.904021:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077921.904031:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077921.904080:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961473792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077921.904119:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077921.904123:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077921.904125:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077921.904131:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3604 00000100:00080000:1.0:1769077921.904135:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077921.904141:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077921.904153:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077921.904155:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961473792:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769077921.904171:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961473920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077921.904185:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077921.904189:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077921.904191:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077921.904233:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3604 00000100:00080000:1.0:1769077921.904238:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077921.904244:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:0.0:1769077921.904251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:1.0:1769077921.904254:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077921.904273:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961474048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077921.904309:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077921/real 1769077921] req@ffff895e4713ed80 x1855008961473792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077921.904339:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961473792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077921.904349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077921.904353:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077921.904356:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961473792:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077921.904383:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961473920:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077921.904407:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077921.904423:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077921/real 1769077921] req@ffff895e4713d880 x1855008961473920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077921.904437:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961473920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077921.904447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077921.904450:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077921.904454:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961473920:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077921.904461:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961474048:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077921.904483:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077921.904496:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077921/real 1769077921] req@ffff895e4713e300 x1855008961474048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077921.904505:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961474048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077921.904521:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077921.904524:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077921.904527:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961474048:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077922.416062:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077922.607931:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077923.439856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077923.631912:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077924.464484:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077924.655870:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077925.487885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077925.679951:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077926.511881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077926.703971:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077927.023845:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077927.023854:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077927.023857:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077927.023865:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3609 00000100:00080000:2.0:1769077927.023876:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077927.023895:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077927.023935:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961474176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077927.023965:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077927.023969:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077927.023971:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077927.023976:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3609 00000100:00080000:2.0:1769077927.023980:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077927.023983:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077927.023993:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077927.024005:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961474304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077927.024036:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077927.024039:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077927.024041:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077927.024046:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3609 00000100:00080000:2.0:1769077927.024052:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077927.024057:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077927.024063:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077927.024071:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961474176:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769077927.024084:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961474432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077927.024179:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077927.024231:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077927/real 1769077927] req@ffff895e44e2a300 x1855008961474176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077927.024249:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961474176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077927.024258:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077927.024262:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077927.024265:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961474176:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077927.024272:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961474304:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077927.024291:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077927.024300:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077927/real 1769077927] req@ffff895e44e2b100 x1855008961474304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077927.024308:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961474304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077927.024326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077927.024329:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077927.024332:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961474304:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077927.024360:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961474432:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077927.024375:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077927.024382:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077927/real 1769077927] req@ffff895e44e2aa00 x1855008961474432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077927.024390:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961474432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077927.024396:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077927.024398:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077927.024400:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961474432:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077927.535884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077927.727890:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077928.559907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077928.751984:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077929.583935:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077929.775963:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077930.607926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077930.799948:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077931.631908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077931.823916:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077932.143943:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077932.143955:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077932.143960:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077932.143969:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3614 00000100:00080000:3.0:1769077932.143981:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077932.143989:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077932.144023:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961474560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077932.144052:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077932.144056:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077932.144058:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077932.144062:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3614 00000100:00080000:3.0:1769077932.144066:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077932.144071:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077932.144080:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077932.144094:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961474688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077932.144105:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077932.144107:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077932.144110:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077932.144113:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961474560:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077932.144115:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3614 00000100:00080000:3.0:1769077932.144122:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077932.144126:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077932.144145:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077932.144160:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961474816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077932.144222:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961474688:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077932.144260:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961474816:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077932.655857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077932.847937:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077933.679861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077933.871890:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077934.703885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077934.895906:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769077935.216029:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769077935.216076:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769077935.216095:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077935.216143:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769077935.216153:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769077935.216247:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077932/real 1769077935] req@ffff895e453d6300 x1855008961474688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077935.216276:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961474688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077935.216289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077935.216301:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077935.216306:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961474688:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077935.216314:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077932/real 1769077935] req@ffff895e453d4380 x1855008961474816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077935.216326:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961474816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077935.216334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077935.216338:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077935.216341:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961474816:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077935.216348:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077932/real 1769077935] req@ffff895e453d6680 x1855008961474560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077935.216361:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961474560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077935.216370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077935.216374:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077935.216395:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961474560:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077935.727846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077935.919921:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769077936.751870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077936.943894:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077937.263855:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077937.263866:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077937.263871:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077937.263880:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3619 00000100:00080000:3.0:1769077937.263892:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077937.263899:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077937.263933:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961474944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077937.263971:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077937.263975:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077937.263977:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077937.263981:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3619 00000100:00080000:3.0:1769077937.263986:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077937.263990:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077937.263999:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077937.264013:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961475072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077937.264026:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961474944:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077937.264028:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077937.264037:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077937.264039:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077937.264043:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3619 00000100:00080000:3.0:1769077937.264063:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077937.264069:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077937.264074:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077937.264088:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961475200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077937.264111:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077937.264162:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077937/real 1769077937] req@ffff895e453d6d80 x1855008961474944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077937.264188:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961474944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077937.264200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077937.264206:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077937.264210:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961474944:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077937.264233:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961475072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077937.264259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077937.264272:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077937/real 1769077937] req@ffff895e453d6680 x1855008961475072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077937.264283:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961475072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077937.264291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077937.264295:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077937.264298:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961475072:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077937.264304:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961475200:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077937.264334:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077937.264345:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077937/real 1769077937] req@ffff895e453d4380 x1855008961475200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077937.264358:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961475200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077937.264367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077937.264371:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077937.264374:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961475200:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077937.775854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077937.967920:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077938.799858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077938.991916:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077939.823828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077940.015916:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077940.847849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077941.039913:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077941.871850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077942.063927:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077942.383846:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077942.383854:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077942.383858:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077942.383873:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3624 00000100:00080000:0.0:1769077942.383883:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077942.383888:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077942.383920:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961475328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077942.383948:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077942.383950:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077942.383952:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077942.383954:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3624 00000100:00080000:0.0:1769077942.383957:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077942.383960:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077942.383968:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077942.383981:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961475328:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077942.383995:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961475456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077942.384004:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077942.384007:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077942.384008:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077942.384011:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3624 00000100:00080000:0.0:1769077942.384014:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077942.384017:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077942.384020:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077942.384027:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961475584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077942.384045:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077942.384082:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077942/real 1769077942] req@ffff895e4383d180 x1855008961475328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077942.384096:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961475328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077942.384106:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077942.384111:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077942.384115:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961475328:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077942.384139:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961475456:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077942.384162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077942.384172:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077942/real 1769077942] req@ffff895e4383df80 x1855008961475456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077942.384191:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961475456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077942.384198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077942.384200:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077942.384202:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961475456:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077942.384207:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961475584:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077942.384221:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077942.384227:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077942/real 1769077942] req@ffff895e4383ed80 x1855008961475584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077942.384235:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961475584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077942.384240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077942.384242:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077942.384244:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961475584:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077942.895888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077943.087914:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077943.919884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077944.111922:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077944.943882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077945.135961:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077945.967904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077946.159977:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077946.991857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077947.183932:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077947.503935:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077947.503949:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077947.503955:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077947.503968:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3630 00000100:00080000:2.0:1769077947.503992:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077947.504001:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077947.504051:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961475712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077947.504086:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077947.504091:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077947.504093:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077947.504098:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3630 00000100:00080000:2.0:1769077947.504102:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077947.504107:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077947.504139:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077947.504159:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961475840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077947.504160:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961475712:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077947.504181:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077947.504186:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077947.504188:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077947.504193:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3630 00000100:00080000:2.0:1769077947.504198:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077947.504203:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077947.504209:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077947.504223:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961475968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077947.504251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077947.504300:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077947/real 1769077947] req@ffff895e44e2b480 x1855008961475712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077947.504336:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961475712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077947.504348:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077947.504353:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077947.504357:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961475712:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077947.504390:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961475840:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077947.504416:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077947.504427:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077947/real 1769077947] req@ffff895e44e28e00 x1855008961475840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077947.504437:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961475840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077947.504443:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077947.504445:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077947.504448:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961475840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077947.504452:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961475968:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077947.504465:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077947.504471:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077947/real 1769077947] req@ffff895e44e29f80 x1855008961475968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077947.504479:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961475968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077947.504493:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077947.504496:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077947.504498:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961475968:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077948.015919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077948.207962:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077949.039865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077949.231926:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077950.063896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077950.255963:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077951.087889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077951.279896:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077952.111888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077952.303943:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077952.623916:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077952.623929:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077952.623933:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077952.623945:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3635 00000100:00080000:1.0:1769077952.623957:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077952.623965:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077952.624010:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961476096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077952.624040:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077952.624043:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077952.624044:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077952.624047:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3635 00000100:00080000:1.0:1769077952.624050:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077952.624053:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077952.624062:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077952.624092:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961476224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077952.624101:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961476096:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077952.624104:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077952.624111:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077952.624112:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077952.624116:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3635 00000100:00080000:1.0:1769077952.624119:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077952.624122:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077952.624126:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077952.624135:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961476352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077952.624192:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077952.624235:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077952/real 1769077952] req@ffff895e4713f100 x1855008961476096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077952.624251:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961476096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077952.624260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077952.624264:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077952.624267:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961476096:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077952.624292:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961476224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077952.624307:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077952.624316:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077952/real 1769077952] req@ffff895e4713ea00 x1855008961476224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077952.624324:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961476224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077952.624338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077952.624341:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077952.624343:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961476224:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077952.624347:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961476352:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077952.624361:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077952.624368:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077952/real 1769077952] req@ffff895e4713d500 x1855008961476352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077952.624375:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961476352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077952.624381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077952.624382:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077952.624384:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961476352:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077953.135890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077953.327968:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077954.159917:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077954.351948:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077955.183850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077955.375964:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077956.207846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077956.399884:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077957.231815:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077957.423917:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077957.743881:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077957.743890:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077957.743894:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077957.743902:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3640 00000100:00080000:0.0:1769077957.743911:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077957.743917:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077957.743948:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961476480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077957.743971:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077957.743974:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077957.743975:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077957.743995:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3640 00000100:00080000:0.0:1769077957.743999:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077957.744003:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077957.744011:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077957.744018:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961476480:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077957.744023:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961476608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077957.744033:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077957.744035:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077957.744037:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077957.744040:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3640 00000100:00080000:0.0:1769077957.744042:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077957.744045:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077957.744048:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077957.744055:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961476736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077957.744090:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077957.744121:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077957/real 1769077957] req@ffff895e4383ed80 x1855008961476480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077957.744135:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961476480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077957.744153:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077957.744156:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077957.744159:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961476480:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077957.744182:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961476608:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077957.744207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077957.744221:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077957/real 1769077957] req@ffff895e4383df80 x1855008961476608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077957.744233:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961476608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077957.744242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077957.744244:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077957.744246:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961476608:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077957.744251:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961476736:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077957.744266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077957.744274:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077957/real 1769077957] req@ffff895e4383d180 x1855008961476736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077957.744281:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961476736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077957.744286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077957.744288:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077957.744290:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961476736:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077958.255900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077958.447972:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077959.279837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077959.471900:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077960.303874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077960.495904:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077961.327843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077961.519920:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077962.351831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077962.543934:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769077962.863878:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077962.863886:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769077962.863890:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077962.863898:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3645 00000100:00080000:2.0:1769077962.863907:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077962.863913:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769077962.863945:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961476864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769077962.863969:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077962.863972:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769077962.863973:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077962.863976:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3645 00000100:00080000:2.0:1769077962.863993:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077962.863996:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077962.864004:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077962.864015:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961476992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769077962.864022:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961476864:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769077962.864026:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769077962.864030:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769077962.864031:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769077962.864034:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3645 00000100:00080000:2.0:1769077962.864036:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769077962.864039:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769077962.864042:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769077962.864050:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961477120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077962.864083:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077962.864121:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077962/real 1769077962] req@ffff895e44e29500 x1855008961476864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077962.864137:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961476864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077962.864145:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077962.864148:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077962.864151:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961476864:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077962.864171:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961476992:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077962.864186:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077962.864194:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077962/real 1769077962] req@ffff895e44e28a80 x1855008961476992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077962.864202:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961476992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077962.864208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077962.864210:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077962.864212:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961476992:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077962.864223:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961477120:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077962.864237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077962.864243:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077962/real 1769077962] req@ffff895e44e29180 x1855008961477120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077962.864251:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961477120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077962.864265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077962.864268:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077962.864270:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961477120:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077963.375913:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077963.568080:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077964.399875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077964.591897:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077965.423851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077965.615909:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077966.447854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077966.639862:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077967.471898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077967.663917:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769077967.983847:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077967.983855:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769077967.983865:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077967.983874:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3650 00000100:00080000:0.0:1769077967.983883:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077967.983889:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769077967.983932:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961477248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077967.983982:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077967.983985:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769077967.983987:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077967.983991:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3650 00000100:00080000:0.0:1769077967.983996:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077967.984001:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077967.984011:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077967.984024:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961477248:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077967.984028:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961477376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769077967.984042:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769077967.984045:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769077967.984047:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769077967.984051:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3650 00000100:00080000:0.0:1769077967.984055:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769077967.984060:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769077967.984065:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077967.984078:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961477504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769077967.984102:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077967.984138:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077967/real 1769077967] req@ffff895e4383d880 x1855008961477248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077967.984157:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961477248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077967.984165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077967.984168:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077967.984171:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961477248:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077967.984217:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961477376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077967.984236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077967.984246:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077967/real 1769077967] req@ffff895e4383e680 x1855008961477376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077967.984263:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961477376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077967.984269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077967.984271:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077967.984273:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961477376:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077967.984278:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961477504:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769077967.984291:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077967.984297:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077967/real 1769077967] req@ffff895e4383c700 x1855008961477504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769077967.984304:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961477504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769077967.984309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769077967.984311:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769077967.984313:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961477504:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077968.495826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077968.687914:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077969.519854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077969.711946:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077970.543848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077970.735924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077971.567826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077971.759956:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077972.591858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077972.784008:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077973.103867:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077973.103876:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077973.103880:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077973.103888:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3655 00000100:00080000:3.0:1769077973.103899:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077973.103905:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077973.103937:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961477632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077973.103981:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077973.103984:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077973.103986:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077973.103989:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3655 00000100:00080000:3.0:1769077973.103993:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077973.103999:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077973.104009:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077973.104023:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961477760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077973.104029:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961477632:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077973.104034:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077973.104044:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077973.104046:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077973.104051:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3655 00000100:00080000:3.0:1769077973.104056:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077973.104062:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077973.104067:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077973.104083:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961477888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077973.104139:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077973.104194:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077973/real 1769077973] req@ffff895e453d6300 x1855008961477632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077973.104231:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961477632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077973.104241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077973.104245:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077973.104248:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961477632:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077973.104274:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961477760:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077973.104293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077973.104302:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077973/real 1769077973] req@ffff895e453d5180 x1855008961477760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077973.104310:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961477760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077973.104316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077973.104319:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077973.104321:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961477760:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077973.104325:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961477888:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077973.104336:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077973.104342:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077973/real 1769077973] req@ffff895e453d5880 x1855008961477888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077973.104350:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961477888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769077998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077973.104364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077973.104366:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077973.104369:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961477888:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077973.615857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077973.807943:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077974.639838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077974.831927:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077975.663870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077975.855922:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077976.687855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077976.879878:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077977.711841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077977.903915:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077978.223855:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077978.223864:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077978.223868:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077978.223875:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3660 00000100:00080000:3.0:1769077978.223901:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077978.223907:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077978.223936:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961478016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077978.223964:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077978.223966:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077978.223968:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077978.223971:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3660 00000100:00080000:3.0:1769077978.223973:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077978.223975:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077978.223982:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077978.223991:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961478144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077978.223998:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077978.224001:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077978.224002:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077978.224005:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3660 00000100:00080000:3.0:1769077978.224007:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077978.224009:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077978.224013:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769077978.224018:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961478016:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769077978.224043:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961478272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077978.224115:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077978.224157:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077978/real 1769077978] req@ffff895e453d4700 x1855008961478016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077978.224171:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961478016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077978.224179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077978.224182:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077978.224185:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961478016:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077978.224190:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961478144:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077978.224206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077978.224214:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077978/real 1769077978] req@ffff895e453d7100 x1855008961478144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077978.224222:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961478144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077978.224228:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077978.224230:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077978.224232:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961478144:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077978.224258:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961478272:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077978.224281:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077978.224289:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077978/real 1769077978] req@ffff895e453d7b80 x1855008961478272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077978.224297:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961478272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077978.224302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077978.224304:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077978.224306:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961478272:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077978.735851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077978.927902:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077979.759870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077979.951921:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077980.783861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077980.975919:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077981.807854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077981.999914:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077982.831857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077983.023921:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077983.343827:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077983.343847:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077983.343851:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077983.343858:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3665 00000100:00080000:3.0:1769077983.343865:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077983.343870:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077983.343894:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961478400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077983.343914:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077983.343916:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077983.343918:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077983.343920:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3665 00000100:00080000:3.0:1769077983.343922:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077983.343924:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077983.343930:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077983.343951:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961478528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077983.343959:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077983.343962:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077983.343963:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769077983.343968:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961478400:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077983.343969:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3665 00000100:00080000:3.0:1769077983.343975:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077983.343978:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077983.343982:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077983.343991:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961478656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077983.344026:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077983.344054:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077983/real 1769077983] req@ffff895e453d5500 x1855008961478400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077983.344080:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961478400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077983.344090:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077983.344094:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077983.344098:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961478400:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077983.344117:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961478528:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077983.344134:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077983.344141:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077983/real 1769077983] req@ffff895e453d4a80 x1855008961478528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077983.344155:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961478528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077983.344161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077983.344163:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077983.344165:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961478528:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077983.344168:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961478656:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077983.344180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077983.344186:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077983/real 1769077983] req@ffff895e453d7480 x1855008961478656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077983.344192:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961478656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077983.344196:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077983.344198:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077983.344200:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961478656:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077983.855844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077984.047970:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077984.879858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077985.072066:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077985.903914:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077986.095936:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077986.927850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077987.119979:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077987.951844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077988.143888:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769077988.463889:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077988.463898:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769077988.463901:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077988.463909:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3671 00000100:00080000:1.0:1769077988.463917:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077988.463921:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769077988.463949:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961478784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077988.463976:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769077988.463980:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769077988.463982:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077988.463985:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3671 00000100:00080000:1.0:1769077988.463989:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077988.463993:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077988.464001:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077988.464011:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961478912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769077988.464019:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769077988.464021:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961478784:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769077988.464034:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769077988.464036:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769077988.464039:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3671 00000100:00080000:1.0:1769077988.464042:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769077988.464045:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769077988.464049:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769077988.464057:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961479040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769077988.464082:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077988.464118:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077988/real 1769077988] req@ffff895e4713ce00 x1855008961478784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077988.464136:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961478784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077988.464157:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077988.464161:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077988.464166:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961478784:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077988.464186:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961478912:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077988.464208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077988.464221:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077988/real 1769077988] req@ffff895e4713f480 x1855008961478912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077988.464243:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961478912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077988.464252:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077988.464255:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077988.464259:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961478912:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077988.464264:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961479040:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077988.464282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077988.464292:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077988/real 1769077988] req@ffff895e4713ca80 x1855008961479040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077988.464304:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961479040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077988.464312:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077988.464315:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077988.464318:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961479040:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769077988.975888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077989.167918:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077989.999856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077990.191879:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077991.023872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077991.215911:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077992.047906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077992.239917:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769077993.071871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077993.263941:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077993.583948:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077993.583956:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077993.583960:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077993.583968:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3676 00000100:00080000:3.0:1769077993.583978:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077993.583983:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077993.584011:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961479168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077993.584033:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077993.584035:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077993.584037:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077993.584039:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3676 00000100:00080000:3.0:1769077993.584041:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077993.584043:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077993.584050:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077993.584063:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961479296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077993.584077:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961479168:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077993.584096:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077993.584100:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077993.584102:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077993.584107:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3676 00000100:00080000:3.0:1769077993.584111:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077993.584115:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077993.584121:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769077993.584132:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769077993.584136:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961479424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077993.584173:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077993/real 1769077993] req@ffff895e453d4e00 x1855008961479168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077993.584192:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961479168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077993.584201:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077993.584205:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077993.584209:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961479168:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769077993.584238:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961479296:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077993.584263:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077993.584275:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077993/real 1769077993] req@ffff895e453d5c00 x1855008961479296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077993.584286:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961479296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077993.584294:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077993.584297:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077993.584300:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961479296:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077993.584307:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961479424:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769077993.584324:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769077993.584334:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077993/real 1769077993] req@ffff895e453d7800 x1855008961479424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769077993.584350:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961479424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769077993.584355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769077993.584357:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769077993.584359:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961479424:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077994.095899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077994.287951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077995.119851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077995.312023:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077996.143811:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077996.335923:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077997.167866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077997.359908:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769077998.191843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077998.383906:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769077998.703839:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077998.703847:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769077998.703850:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077998.703857:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3681 00000100:00080000:3.0:1769077998.703866:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077998.703870:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769077998.703902:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961479552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769077998.703927:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077998.703929:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769077998.703930:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077998.703933:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3681 00000100:00080000:3.0:1769077998.703935:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077998.703938:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077998.703956:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077998.703967:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961479680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077998.703976:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961479552:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769077998.703977:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769077998.703984:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769077998.703986:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769077998.703989:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3681 00000100:00080000:3.0:1769077998.703991:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769077998.703994:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769077998.703997:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769077998.704004:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961479808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769077998.704065:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961479680:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769077998.704084:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961479808:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769077999.215843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769077999.407863:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078000.239844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078000.431898:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078001.263848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078001.455898:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769078001.776015:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769078001.776053:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769078001.776069:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769078001.776102:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769078001.776110:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769078001.776167:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077998/real 1769078001] req@ffff895e453d7800 x1855008961479680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078001.776190:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961479680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078001.776201:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078001.776206:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078001.776209:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961479680:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078001.776216:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077998/real 1769078001] req@ffff895e453d5c00 x1855008961479808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078001.776227:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961479808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078001.776253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078001.776257:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078001.776261:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961479808:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078001.776267:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769077998/real 1769078001] req@ffff895e453d5f80 x1855008961479552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078001.776280:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961479552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078001.776287:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078001.776291:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078001.776294:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961479552:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078002.287881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078002.479962:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078003.311889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078003.503922:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078003.823832:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078003.823840:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078003.823843:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078003.823851:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3686 00000100:00080000:3.0:1769078003.823860:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078003.823865:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078003.823892:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961479936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078003.823920:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078003.823923:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078003.823925:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078003.823929:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3686 00000100:00080000:3.0:1769078003.823945:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078003.823948:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078003.823956:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078003.823966:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961479936:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078003.823968:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961480064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078003.823982:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078003.823985:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078003.823987:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078003.823991:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3686 00000100:00080000:3.0:1769078003.823995:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078003.824000:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078003.824005:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078003.824017:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961480192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078003.824038:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078003.824075:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078003/real 1769078003] req@ffff895e453d4e00 x1855008961479936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078003.824095:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961479936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078003.824104:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078003.824107:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078003.824111:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961479936:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078003.824134:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961480064:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078003.824164:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078003.824177:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078003/real 1769078003] req@ffff895e453d7480 x1855008961480064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078003.824188:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961480064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078003.824195:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078003.824199:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078003.824202:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961480064:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078003.824209:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961480192:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078003.824245:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078003.824255:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078003/real 1769078003] req@ffff895e453d4a80 x1855008961480192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078003.824266:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961480192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078003.824274:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078003.824277:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078003.824280:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961480192:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078004.335842:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078004.527895:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078005.359854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078005.551854:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078006.383834:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078006.575893:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078007.407855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078007.599876:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078008.431840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078008.623894:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078008.943854:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078008.943862:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078008.943865:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078008.943872:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3691 00000100:00080000:2.0:1769078008.943881:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078008.943887:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078008.943922:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961480320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078008.943946:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078008.943967:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078008.943969:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078008.943973:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3691 00000100:00080000:2.0:1769078008.943977:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078008.943981:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078008.943984:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961480320:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769078008.943989:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078008.944000:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961480448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078008.944010:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078008.944013:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078008.944015:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078008.944018:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3691 00000100:00080000:2.0:1769078008.944021:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078008.944026:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078008.944030:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078008.944039:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961480576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078008.944063:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078008.944104:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078008/real 1769078008] req@ffff895e44e2bb80 x1855008961480320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078008.944117:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961480320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078008.944124:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078008.944128:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078008.944130:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961480320:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078008.944149:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961480448:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078008.944163:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078008.944171:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078008/real 1769078008] req@ffff895e44e2b800 x1855008961480448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078008.944178:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961480448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078008.944183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078008.944184:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078008.944186:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961480448:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078008.944189:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961480576:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078008.944198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078008.944203:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078008/real 1769078008] req@ffff895e44e29c00 x1855008961480576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078008.944209:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961480576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078008.944222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078008.944224:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078008.944226:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961480576:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078009.455824:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078009.647860:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078010.479855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078010.671924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078011.503875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078011.695917:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078012.527837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078012.719874:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078013.551823:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078013.743905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078014.063859:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078014.063868:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078014.063871:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078014.063879:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3696 00000100:00080000:2.0:1769078014.063889:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078014.063912:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078014.063942:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961480704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078014.063966:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078014.063969:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078014.063970:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078014.063973:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3696 00000100:00080000:2.0:1769078014.063976:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078014.063986:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078014.063993:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078014.064002:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961480832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078014.064010:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961480704:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078014.064013:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078014.064019:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078014.064020:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078014.064023:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3696 00000100:00080000:2.0:1769078014.064025:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078014.064028:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078014.064032:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078014.064040:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961480960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078014.064079:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078014.064141:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078014/real 1769078014] req@ffff895e44e29c00 x1855008961480704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078014.064161:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961480704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078014.064169:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078014.064172:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078014.064175:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961480704:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078014.064196:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961480832:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078014.064224:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078014.064233:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078014/real 1769078014] req@ffff895e44e2b800 x1855008961480832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078014.064242:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961480832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078014.064248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078014.064250:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078014.064252:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961480832:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078014.064264:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961480960:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078014.064278:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078014.064285:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078014/real 1769078014] req@ffff895e44e2bb80 x1855008961480960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078014.064293:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961480960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078014.064298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078014.064300:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078014.064302:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961480960:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078014.575919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078014.767892:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078015.599851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078015.792056:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078016.623884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078016.815907:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078017.647867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078017.839884:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078018.671867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078018.863911:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078019.183875:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078019.183883:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078019.183903:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078019.183911:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3701 00000100:00080000:3.0:1769078019.183920:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078019.183926:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078019.183953:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961481088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078019.183982:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078019.183985:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078019.183988:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078019.183992:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3701 00000100:00080000:3.0:1769078019.183995:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078019.184000:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078019.184015:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078019.184030:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961481216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078019.184031:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961481088:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078019.184041:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078019.184045:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078019.184046:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078019.184050:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3701 00000100:00080000:3.0:1769078019.184053:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078019.184057:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078019.184071:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078019.184080:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961481344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078019.184091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078019.184128:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078019/real 1769078019] req@ffff895e453d5500 x1855008961481088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078019.184147:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961481088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078019.184155:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078019.184158:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078019.184160:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961481088:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078019.184180:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961481216:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078019.184199:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078019.184207:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078019/real 1769078019] req@ffff895e453d7b80 x1855008961481216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078019.184228:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961481216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078019.184237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078019.184240:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078019.184244:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961481216:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078019.184250:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961481344:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078019.184273:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078019.184284:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078019/real 1769078019] req@ffff895e453d7100 x1855008961481344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078019.184292:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961481344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078019.184297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078019.184299:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078019.184301:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961481344:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078019.695834:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078019.887884:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078020.719860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078020.911864:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078021.743853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078021.935912:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078022.767852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078022.960004:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078023.791885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078023.983912:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078024.303821:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078024.303833:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078024.303838:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078024.303849:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3706 00000100:00080000:3.0:1769078024.303862:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078024.303872:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078024.303923:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961481472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078024.303960:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078024.303965:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078024.303968:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078024.303974:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3706 00000100:00080000:3.0:1769078024.303979:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078024.303985:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078024.303997:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078024.304005:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961481472:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078024.304016:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961481600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078024.304030:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078024.304033:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078024.304036:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078024.304041:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3706 00000100:00080000:3.0:1769078024.304074:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769078024.304081:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769078024.304082:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078024.304091:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078024.304107:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961481728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078024.304127:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078024/real 1769078024] req@ffff895e453d4700 x1855008961481472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078024.304144:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961481472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078024.304164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078024.304168:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078024.304171:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961481472:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078024.304195:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961481600:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078024.304212:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078024.304221:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078024/real 1769078024] req@ffff895e453d5880 x1855008961481600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078024.304233:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961481600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078024.304243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078024.304276:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078024.304280:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961481600:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078024.304288:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961481728:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078024.304311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078024.304335:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078024/real 1769078024] req@ffff895e453d5180 x1855008961481728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078024.304348:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961481728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078024.304357:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078024.304361:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078024.304364:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961481728:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078024.815859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078025.007911:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078025.839878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078026.031938:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078026.863885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078027.055872:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078027.887902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078028.079977:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078028.911927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078029.103905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078029.423907:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078029.423921:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078029.423927:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078029.423939:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3711 00000100:00080000:2.0:1769078029.423954:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078029.423964:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078029.424015:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961481856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078029.424043:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078029.424046:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078029.424048:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078029.424051:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3711 00000100:00080000:2.0:1769078029.424054:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078029.424057:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078029.424066:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078029.424076:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961481984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078029.424085:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078029.424088:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769078029.424101:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961481856:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078029.424107:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078029.424112:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3711 00000100:00080000:2.0:1769078029.424115:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078029.424118:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078029.424123:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078029.424134:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961482112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078029.424188:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078029.424230:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078029/real 1769078029] req@ffff895e44e29180 x1855008961481856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078029.424246:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961481856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078029.424254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078029.424258:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078029.424261:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961481856:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078029.424269:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961481984:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078029.424293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078029.424306:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078029/real 1769078029] req@ffff895e44e28a80 x1855008961481984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078029.424318:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961481984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078029.424327:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078029.424331:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078029.424334:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961481984:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078029.424364:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961482112:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078029.424380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078029.424387:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078029/real 1769078029] req@ffff895e44e29500 x1855008961482112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078029.424403:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961482112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078029.424418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078029.424420:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078029.424422:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961482112:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078029.935874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078030.127920:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078030.959830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078031.151846:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078031.983853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078032.175912:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078033.007855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078033.199872:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078034.031852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078034.223905:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078034.543863:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078034.543872:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078034.543876:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078034.543883:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3717 00000100:00080000:2.0:1769078034.543893:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078034.543933:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078034.543972:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961482240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078034.544000:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078034.544003:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078034.544005:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078034.544008:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3717 00000100:00080000:2.0:1769078034.544011:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078034.544013:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078034.544020:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078034.544040:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961482368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078034.544047:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078034.544049:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078034.544052:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1769078034.544053:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961482240:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078034.544055:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3717 00000100:00080000:2.0:1769078034.544059:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078034.544063:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078034.544066:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078034.544088:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961482496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078034.544130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078034.544167:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078034/real 1769078034] req@ffff895e44e29f80 x1855008961482240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078034.544192:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961482240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078034.544201:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078034.544205:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078034.544208:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961482240:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078034.544242:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961482368:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078034.544266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078034.544278:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078034/real 1769078034] req@ffff895e44e28e00 x1855008961482368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078034.544289:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961482368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078034.544307:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078034.544311:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078034.544314:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961482368:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078034.544320:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961482496:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078034.544341:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078034.544352:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078034/real 1769078034] req@ffff895e44e2b480 x1855008961482496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078034.544362:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961482496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078034.544369:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078034.544372:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078034.544375:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961482496:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078035.055844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078035.247918:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078036.079844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078036.271870:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078037.103813:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078037.295911:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078038.127855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078038.319903:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078039.151921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078039.343916:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078039.663868:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078039.663878:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078039.663882:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078039.663892:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3722 00000100:00080000:2.0:1769078039.663901:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078039.663907:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078039.663950:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961482624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078039.663976:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078039.663980:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078039.664008:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078039.664013:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3722 00000100:00080000:2.0:1769078039.664018:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078039.664022:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078039.664032:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078039.664046:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961482752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078039.664053:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961482624:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078039.664074:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078039.664078:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078039.664080:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078039.664086:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3722 00000100:00080000:2.0:1769078039.664090:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078039.664095:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078039.664101:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078039.664117:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961482880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078039.664141:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078039.664184:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078039/real 1769078039] req@ffff895e44e2aa00 x1855008961482624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078039.664198:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961482624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078039.664216:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078039.664219:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078039.664231:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961482624:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078039.664257:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961482752:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078039.664275:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078039.664283:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078039/real 1769078039] req@ffff895e44e2b100 x1855008961482752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078039.664291:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961482752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078039.664297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078039.664299:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078039.664301:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961482752:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078039.664305:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961482880:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078039.664315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078039.664320:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078039/real 1769078039] req@ffff895e44e2a300 x1855008961482880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078039.664327:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961482880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078039.664332:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078039.664333:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078039.664335:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961482880:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078040.175989:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078040.367899:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078041.199896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078041.391919:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078042.223845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078042.415893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078043.247877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078043.439894:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078044.271894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078044.463877:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078044.783898:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078044.783939:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078044.783944:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078044.783956:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3727 00000100:00080000:1.0:1769078044.783969:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078044.783977:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078044.784022:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961483008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078044.784056:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078044.784060:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078044.784062:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078044.784066:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3727 00000100:00080000:1.0:1769078044.784070:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078044.784075:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078044.784087:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078044.784100:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961483008:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078044.784103:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961483136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078044.784144:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078044.784147:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078044.784157:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078044.784162:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3727 00000100:00080000:1.0:1769078044.784167:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1769078044.784168:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:1.0:1769078044.784174:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078044.784182:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078044.784196:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961483264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078044.784224:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078044/real 1769078044] req@ffff895e4713f800 x1855008961483008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078044.784241:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961483008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078044.784249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078044.784253:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078044.784255:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961483008:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078044.784278:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961483136:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078044.784294:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078044.784302:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078044/real 1769078044] req@ffff895e4713dc00 x1855008961483136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078044.784309:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961483136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078044.784315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078044.784317:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078044.784319:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961483136:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078044.784322:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961483264:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078044.784330:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078044.784335:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078044/real 1769078044] req@ffff895e4713df80 x1855008961483264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078044.784342:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961483264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078044.784355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078044.784357:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078044.784359:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961483264:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078045.295867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078045.487938:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078046.319870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078046.511902:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078047.343903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078047.535951:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078048.367855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078048.559896:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078049.391878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078049.583978:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078049.903919:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078049.903929:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078049.903934:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078049.903944:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3732 00000100:00080000:2.0:1769078049.903957:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078049.903963:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078049.904012:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961483392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078049.904053:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078049.904058:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078049.904060:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078049.904064:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3732 00000100:00080000:2.0:1769078049.904068:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078049.904071:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078049.904082:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078049.904095:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961483520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078049.904097:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961483392:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078049.904105:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078049.904108:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078049.904110:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078049.904113:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3732 00000100:00080000:2.0:1769078049.904134:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078049.904138:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078049.904142:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078049.904151:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961483648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078049.904173:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078049.904219:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078049/real 1769078049] req@ffff895e44e28000 x1855008961483392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078049.904239:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961483392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078049.904246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078049.904249:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078049.904252:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961483392:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078049.904276:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961483520:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078049.904292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078049.904300:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078049/real 1769078049] req@ffff895e44e28380 x1855008961483520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078049.904307:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961483520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078049.904313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078049.904315:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078049.904317:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961483520:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078049.904326:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961483648:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078049.904339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078049.904345:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078049/real 1769078049] req@ffff895e44e29880 x1855008961483648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078049.904352:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961483648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078049.904356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078049.904358:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078049.904360:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961483648:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078050.415884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078050.607953:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078051.439880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078051.631872:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078052.463883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078052.655927:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078053.487867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078053.679930:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078054.511804:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078054.703850:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078055.023869:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078055.023881:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078055.023886:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078055.023897:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3737 00000100:00080000:2.0:1769078055.023908:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078055.023915:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078055.023956:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961483776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078055.023987:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078055.023992:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078055.023994:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078055.023999:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3737 00000100:00080000:2.0:1769078055.024004:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078055.024022:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078055.024032:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078055.024045:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961483904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078055.024056:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078055.024060:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078055.024062:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078055.024067:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3737 00000100:00080000:2.0:1769078055.024072:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078055.024077:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078055.024092:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078055.024102:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961483776:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078055.024106:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961484032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078055.024171:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078055.024212:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078055/real 1769078055] req@ffff895e44e29880 x1855008961483776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078055.024231:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961483776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078055.024242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078055.024247:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078055.024251:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961483776:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078055.024259:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961483904:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078055.024283:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078055.024295:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078055/real 1769078055] req@ffff895e44e28380 x1855008961483904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078055.024315:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961483904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078055.024324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078055.024328:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078055.024331:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961483904:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078055.024355:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961484032:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078055.024375:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078055.024386:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078055/real 1769078055] req@ffff895e44e28000 x1855008961484032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078055.024397:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961484032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078055.024405:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078055.024408:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078055.024411:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961484032:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078055.535873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078055.727983:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078056.559856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078056.751914:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078057.583931:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078057.775861:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078058.607848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078058.799912:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078059.631841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078059.823905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078060.143879:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078060.143890:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078060.143895:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078060.143905:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3742 00000100:00080000:2.0:1769078060.143917:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078060.143925:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078060.143979:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961484160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078060.144009:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078060.144013:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078060.144015:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078060.144020:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3742 00000100:00080000:2.0:1769078060.144025:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078060.144029:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078060.144039:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078060.144053:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961484288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078060.144065:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078060.144070:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769078060.144072:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961484160:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078060.144073:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078060.144078:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3742 00000100:00080000:2.0:1769078060.144086:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078060.144089:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078060.144093:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078060.144103:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961484416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078060.144144:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078060.144236:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078060/real 1769078060] req@ffff895e44e2a300 x1855008961484160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078060.144255:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961484160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078060.144279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078060.144284:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078060.144288:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961484160:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078060.144313:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961484288:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078060.144338:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078060.144351:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078060/real 1769078060] req@ffff895e44e2b100 x1855008961484288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078060.144364:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961484288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078060.144374:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078060.144377:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078060.144380:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961484288:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078060.144387:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961484416:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078060.144407:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078060.144419:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078060/real 1769078060] req@ffff895e44e2aa00 x1855008961484416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078060.144432:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961484416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078060.144440:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078060.144444:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078060.144447:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961484416:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078060.655856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078060.847876:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078061.679846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078061.871906:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078062.703819:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078062.895908:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078063.727833:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078063.919874:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078064.751857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078064.943926:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078065.263835:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078065.263847:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078065.263852:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078065.263862:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3747 00000100:00080000:3.0:1769078065.263874:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078065.263881:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078065.263928:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961484544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078065.263954:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078065.263956:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078065.263958:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078065.263961:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3747 00000100:00080000:3.0:1769078065.263964:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078065.263967:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078065.263974:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078065.263984:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961484672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078065.263993:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078065.263996:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078065.263997:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078065.264001:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3747 00000100:00100000:0.0:1769078065.264002:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961484544:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078065.264004:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078065.264011:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078065.264015:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078065.264038:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961484800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078065.264108:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961484672:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078065.264135:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961484800:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078065.775876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078065.967881:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078066.799843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:3.0:1769078066.991893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078067.823845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078068.015865:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769078068.336227:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769078068.336280:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078068.336301:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078068.336352:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078068.336362:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769078068.336462:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078065/real 1769078068] req@ffff895e453d4380 x1855008961484672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078068.336489:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961484672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078068.336502:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078068.336541:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078068.336546:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961484672:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078068.336554:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078065/real 1769078068] req@ffff895e453d6680 x1855008961484800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078068.336567:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961484800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078068.336576:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078068.336580:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078068.336583:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961484800:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078068.336590:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078065/real 1769078068] req@ffff895e453d6300 x1855008961484544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078068.336602:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961484544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078068.336636:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078068.336640:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078068.336643:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961484544:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078068.847925:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078069.039966:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078069.871873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078070.063914:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078070.383861:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078070.383874:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078070.383879:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078070.383890:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3752 00000100:00080000:3.0:1769078070.383902:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078070.383910:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078070.383946:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961484928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078070.383980:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078070.383983:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078070.383985:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078070.383987:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3752 00000100:00080000:3.0:1769078070.383991:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078070.383994:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078070.384002:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078070.384016:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961485056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078070.384020:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961484928:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078070.384036:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078070.384040:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078070.384042:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078070.384047:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3752 00000100:00080000:3.0:1769078070.384083:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078070.384090:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:0.0:1769078070.384092:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:3.0:1769078070.384098:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078070.384114:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961485184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078070.384136:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078070/real 1769078070] req@ffff895e453d6d80 x1855008961484928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078070.384157:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961484928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078070.384168:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078070.384173:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078070.384178:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961484928:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078070.384202:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961485056:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078070.384233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078070.384249:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078070/real 1769078070] req@ffff895e453d6300 x1855008961485056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078070.384261:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961485056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078070.384269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078070.384272:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078070.384276:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961485056:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078070.384291:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961485184:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078070.384314:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078070.384326:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078070/real 1769078070] req@ffff895e453d6680 x1855008961485184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078070.384350:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961485184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078070.384359:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078070.384362:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078070.384365:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961485184:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078070.895853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078071.087916:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078071.919846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078072.111912:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078072.943843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078073.135909:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078073.967841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078074.159916:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078074.991870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078075.183910:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078075.503862:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078075.503870:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078075.503874:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078075.503882:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3758 00000100:00080000:3.0:1769078075.503891:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078075.503897:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078075.503929:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961485312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078075.503952:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078075.503960:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078075.503962:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078075.503966:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3758 00000100:00080000:3.0:1769078075.503970:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078075.503974:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078075.503985:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078075.503997:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961485312:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078075.504002:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961485440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078075.504015:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078075.504031:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078075.504033:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078075.504036:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3758 00000100:00080000:3.0:1769078075.504040:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078075.504044:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078075.504049:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1769078075.504054:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078075.504067:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961485568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078075.504095:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078075/real 1769078075] req@ffff895e453d4380 x1855008961485312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078075.504125:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961485312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078075.504134:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078075.504138:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078075.504140:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961485312:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078075.504163:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961485440:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078075.504188:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078075.504209:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078075/real 1769078075] req@ffff895e453d5180 x1855008961485440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078075.504228:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961485440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078075.504234:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078075.504236:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078075.504239:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961485440:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078075.504243:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961485568:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078075.504258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078075.504267:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078075/real 1769078075] req@ffff895e453d5880 x1855008961485568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078075.504279:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961485568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078075.504287:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078075.504291:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078075.504295:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961485568:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078076.015819:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078076.207918:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078077.039876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078077.231918:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078078.063843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078078.255901:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078079.087845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078079.279882:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078080.111838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078080.303889:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078080.623822:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078080.623830:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078080.623833:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078080.623842:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3763 00000100:00080000:0.0:1769078080.623852:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078080.623858:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078080.623892:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961485696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078080.623923:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078080.623930:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078080.623932:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078080.623935:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3763 00000100:00080000:0.0:1769078080.623939:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078080.623942:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078080.623949:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078080.623949:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961485696:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078080.623962:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961485824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078080.623970:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078080.623972:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078080.623973:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078080.623976:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3763 00000100:00080000:0.0:1769078080.623978:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078080.623980:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078080.623982:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078080.623990:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961485952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078080.624014:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078080.624041:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078080/real 1769078080] req@ffff895e4383fb80 x1855008961485696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078080.624053:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961485696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078080.624059:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078080.624061:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078080.624070:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961485696:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078080.624087:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961485824:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078080.624100:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078080.624107:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078080/real 1769078080] req@ffff895e4383f100 x1855008961485824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078080.624114:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961485824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078080.624124:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078080.624126:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078080.624128:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961485824:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078080.624131:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961485952:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078080.624143:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078080.624148:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078080/real 1769078080] req@ffff895e4383c380 x1855008961485952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078080.624154:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961485952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078080.624159:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078080.624160:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078080.624162:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961485952:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078081.135857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078081.327873:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078082.159837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078082.351883:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078083.183839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078083.375905:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078084.207841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078084.399861:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078085.231845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078085.423869:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078085.743841:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078085.743849:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078085.743853:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078085.743860:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3768 00000100:00080000:0.0:1769078085.743883:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078085.743888:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078085.743912:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961486080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078085.743933:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078085.743935:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078085.743937:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078085.743939:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3768 00000100:00080000:0.0:1769078085.743941:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078085.743943:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078085.743950:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078085.743958:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961486208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078085.743965:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078085.743967:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078085.743968:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078085.743970:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3768 00000100:00080000:0.0:1769078085.743973:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078085.743975:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078085.743977:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078085.743984:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961486336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078085.743984:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961486080:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078085.744048:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078085.744087:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078085/real 1769078085] req@ffff895e4383c000 x1855008961486080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078085.744105:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961486080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078085.744115:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078085.744119:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078085.744129:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961486080:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078085.744136:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961486208:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078085.744159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078085.744171:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078085/real 1769078085] req@ffff895e4383ce00 x1855008961486208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078085.744183:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961486208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078085.744191:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078085.744194:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078085.744197:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961486208:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078085.744232:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961486336:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078085.744251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078085.744261:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078085/real 1769078085] req@ffff895e4383f480 x1855008961486336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078085.744272:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961486336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078085.744279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078085.744282:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078085.744286:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961486336:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078086.255853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078086.447904:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078087.279838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078087.471875:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078088.303846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078088.495892:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078089.327858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078089.519953:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078090.351886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078090.543898:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078090.863903:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078090.863938:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078090.863943:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078090.863955:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3773 00000100:00080000:0.0:1769078090.863970:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078090.863979:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078090.864021:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961486464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078090.864056:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078090.864060:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078090.864062:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078090.864066:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3773 00000100:00080000:0.0:1769078090.864070:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078090.864075:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078090.864087:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078090.864103:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961486592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078090.864115:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078090.864118:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078090.864120:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078090.864140:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3773 00000100:00080000:0.0:1769078090.864145:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078090.864158:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078090.864159:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961486464:192.168.204.117@tcp:250:kworker.0 00000008:00100000:0.0:1769078090.864165:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078090.864182:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961486720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078090.864230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078090.864277:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078090/real 1769078090] req@ffff895e4383ea00 x1855008961486464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078090.864291:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961486464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078090.864298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078090.864302:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078090.864305:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961486464:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078090.864323:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961486592:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078090.864337:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078090.864345:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078090/real 1769078090] req@ffff895e4383dc00 x1855008961486592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078090.864352:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961486592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078090.864357:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078090.864359:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078090.864361:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961486592:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078090.864365:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961486720:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078090.864374:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078090.864378:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078090/real 1769078090] req@ffff895e4383ca80 x1855008961486720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078090.864385:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961486720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078090.864398:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078090.864400:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078090.864402:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961486720:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078091.375858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078091.567959:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078092.399882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078092.591942:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078093.423871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078093.615940:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078094.447903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078094.639948:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078095.471877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078095.663953:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078095.983894:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078095.983907:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078095.983912:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078095.983925:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3778 00000100:00080000:1.0:1769078095.983939:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078095.983947:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078095.983991:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961486848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078095.984019:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078095.984022:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078095.984024:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078095.984029:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3778 00000100:00080000:1.0:1769078095.984033:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078095.984039:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078095.984082:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961486848:192.168.204.117@tcp:250:kworker.0 00000008:00100000:1.0:1769078095.984103:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078095.984121:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961486976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078095.984135:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078095.984138:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078095.984141:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078095.984145:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3778 00000100:00080000:1.0:1769078095.984151:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078095.984156:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078095.984161:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769078095.984169:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078095.984178:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961487104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078095.984215:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078095/real 1769078095] req@ffff895e4713c380 x1855008961486848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078095.984232:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961486848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078095.984240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078095.984244:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078095.984247:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961486848:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078095.984268:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961486976:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078095.984284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078095.984292:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078095/real 1769078095] req@ffff895e4713d180 x1855008961486976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078095.984300:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961486976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078095.984306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078095.984309:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078095.984311:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961486976:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078095.984322:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961487104:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078095.984336:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078095.984343:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078095/real 1769078095] req@ffff895e4713df80 x1855008961487104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078095.984351:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961487104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078095.984356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078095.984359:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078095.984361:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961487104:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078096.495942:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078096.687901:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078097.519865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078097.711897:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078098.543856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078098.736009:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078099.567846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078099.759954:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078100.591960:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078100.783918:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078101.103887:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078101.103898:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078101.103902:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078101.103909:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3783 00000100:00080000:0.0:1769078101.103919:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078101.103925:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078101.103972:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961487232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078101.103999:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078101.104004:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078101.104006:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078101.104009:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3783 00000100:00080000:0.0:1769078101.104012:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078101.104015:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078101.104024:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078101.104035:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961487360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078101.104043:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078101.104045:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078101.104046:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078101.104052:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3783 00000100:00100000:3.0:1769078101.104054:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961487232:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078101.104056:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078101.104065:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078101.104088:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078101.104103:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961487488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078101.104120:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078101.104157:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078101/real 1769078101] req@ffff895e4383f800 x1855008961487232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078101.104176:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961487232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078101.104187:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078101.104191:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078101.104195:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961487232:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078101.104217:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961487360:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078101.104236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078101.104256:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078101/real 1769078101] req@ffff895e4383e300 x1855008961487360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078101.104269:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961487360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078101.104279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078101.104283:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078101.104287:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961487360:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078101.104293:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961487488:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078101.104310:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078101.104319:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078101/real 1769078101] req@ffff895e4383d500 x1855008961487488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078101.104332:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961487488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078101.104340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078101.104344:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078101.104347:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961487488:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078101.615862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078101.807907:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078102.639843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078102.831988:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078103.663851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078103.855938:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078104.687838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078104.879908:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078105.711835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078105.903952:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078106.223893:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078106.223905:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078106.223931:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078106.223963:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3788 00000100:00080000:0.0:1769078106.223988:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078106.223994:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078106.224058:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961487616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078106.224098:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078106.224101:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078106.224102:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078106.224105:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3788 00000100:00080000:0.0:1769078106.224108:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078106.224111:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078106.224119:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078106.224130:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961487744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078106.224145:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961487616:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078106.224161:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078106.224179:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078106.224180:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078106.224184:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3788 00000800:00000100:3.0:1769078106.224198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769078106.224226:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078106.224231:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078106.224252:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078106.224252:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078106/real 1769078106] req@ffff895e4383d500 x1855008961487616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078106.224275:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961487616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078106.224280:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961487872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078106.224291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078106.224296:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078106.224299:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961487616:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078106.224327:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961487744:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078106.224347:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078106.224358:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078106/real 1769078106] req@ffff895e4383e300 x1855008961487744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078106.224366:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961487744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078106.224373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078106.224376:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078106.224379:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961487744:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078106.224384:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961487872:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078106.224394:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078106.224403:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078106/real 1769078106] req@ffff895e4383f800 x1855008961487872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078106.224410:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961487872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078106.224425:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078106.224428:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078106.224430:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961487872:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078106.735855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078106.927888:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078107.759864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078107.951936:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078108.783847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078108.976002:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078109.807863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078109.999937:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078110.831896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078111.023925:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078111.343908:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078111.343927:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078111.343931:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078111.343941:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3793 00000100:00080000:2.0:1769078111.343953:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078111.343961:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078111.343997:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961488000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078111.344019:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078111.344022:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078111.344036:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078111.344039:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3793 00000100:00080000:2.0:1769078111.344043:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078111.344046:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078111.344054:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078111.344065:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961488128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078111.344076:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769078111.344076:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961488000:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078111.344079:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078111.344083:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078111.344090:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3793 00000100:00080000:2.0:1769078111.344094:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078111.344098:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078111.344112:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078111.344127:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961488256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078111.344144:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078111.344183:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078111/real 1769078111] req@ffff895e44e2b480 x1855008961488000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078111.344203:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961488000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078111.344214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078111.344218:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078111.344223:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961488000:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078111.344249:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961488128:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078111.344275:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078111.344289:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078111/real 1769078111] req@ffff895e44e28e00 x1855008961488128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078111.344304:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961488128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078111.344322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078111.344333:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078111.344337:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961488128:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078111.344344:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961488256:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078111.344370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078111.344383:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078111/real 1769078111] req@ffff895e44e29f80 x1855008961488256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078111.344395:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961488256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078111.344404:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078111.344408:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078111.344411:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961488256:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078111.855862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078112.047931:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078112.879851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078113.071944:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078113.903873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078114.095988:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078114.927854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078115.119996:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078115.951879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078116.143929:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078116.463854:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078116.463862:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078116.463865:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078116.463873:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3799 00000100:00080000:3.0:1769078116.463881:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078116.463886:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078116.463914:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961488384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078116.463950:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078116.463953:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078116.463954:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078116.463957:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3799 00000100:00080000:3.0:1769078116.463959:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078116.463962:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078116.463968:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078116.463977:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961488512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078116.463984:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078116.463986:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078116.463987:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078116.463992:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3799 00000100:00100000:0.0:1769078116.463994:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961488384:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078116.463994:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078116.463999:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078116.464005:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078116.464011:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961488640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078116.464051:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078116.464085:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078116/real 1769078116] req@ffff895e453d4700 x1855008961488384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078116.464099:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961488384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078116.464106:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078116.464109:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078116.464111:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961488384:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078116.464116:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961488512:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078116.464128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078116.464135:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078116/real 1769078116] req@ffff895e453d7100 x1855008961488512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078116.464142:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961488512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078116.464147:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078116.464148:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078116.464155:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961488512:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078116.464172:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961488640:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078116.464184:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078116.464189:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078116/real 1769078116] req@ffff895e453d7b80 x1855008961488640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078116.464196:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961488640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078116.464200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078116.464202:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078116.464204:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961488640:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078116.975860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078117.167911:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078117.999849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078118.191874:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078119.023846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078119.215899:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078120.047826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078120.239889:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078121.071884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078121.263950:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078121.583849:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078121.583861:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078121.583865:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078121.583874:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3804 00000100:00080000:0.0:1769078121.583886:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078121.583892:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078121.583936:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961488768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078121.583965:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078121.583969:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078121.583971:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078121.583976:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3804 00000100:00080000:0.0:1769078121.583980:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078121.583993:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961488768:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078121.584004:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078121.584014:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078121.584026:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961488896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078121.584034:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078121.584037:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078121.584038:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078121.584041:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3804 00000100:00080000:0.0:1769078121.584045:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078121.584049:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078121.584054:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078121.584064:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961489024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078121.584078:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078121.584114:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078121/real 1769078121] req@ffff895e4383ca80 x1855008961488768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078121.584140:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961488768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078121.584148:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078121.584151:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078121.584154:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961488768:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078121.584175:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961488896:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078121.584191:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078121.584204:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078121/real 1769078121] req@ffff895e4383dc00 x1855008961488896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078121.584211:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961488896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078121.584217:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078121.584219:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078121.584220:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961488896:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078121.584224:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961489024:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078121.584235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078121.584240:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078121/real 1769078121] req@ffff895e4383ea00 x1855008961489024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078121.584247:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961489024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078121.584259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078121.584262:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078121.584265:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961489024:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078122.095884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078122.287895:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078123.119909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078123.311966:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078124.143874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078124.335884:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078125.167910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078125.359943:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078126.191893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078126.383880:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078126.703894:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078126.703908:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078126.703914:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078126.703928:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3809 00000100:00080000:2.0:1769078126.703943:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078126.703951:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078126.703999:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961489152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078126.704045:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078126.704049:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078126.704052:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078126.704057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3809 00000100:00080000:2.0:1769078126.704061:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078126.704087:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078126.704100:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078126.704105:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961489152:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078126.704121:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961489280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078126.704138:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078126.704143:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078126.704145:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078126.704150:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3809 00000100:00080000:2.0:1769078126.704172:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078126.704178:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:3.0:1769078126.704186:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:2.0:1769078126.704189:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078126.704208:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961489408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078126.704247:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078126/real 1769078126] req@ffff895e44e29500 x1855008961489152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078126.704267:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961489152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078126.704275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078126.704279:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078126.704282:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961489152:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078126.704310:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961489280:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078126.704330:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078126.704339:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078126/real 1769078126] req@ffff895e44e28a80 x1855008961489280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078126.704347:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961489280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078126.704353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078126.704356:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078126.704358:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961489280:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078126.704377:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961489408:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078126.704391:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078126.704398:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078126/real 1769078126] req@ffff895e44e29180 x1855008961489408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078126.704405:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961489408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078126.704411:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078126.704413:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078126.704415:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961489408:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078127.215901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078127.407967:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078128.239853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078128.431976:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078129.263926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078129.455953:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078130.287822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078130.479929:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078131.311943:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078131.503989:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078131.823904:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078131.823919:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078131.823925:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078131.823937:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3814 00000100:00080000:0.0:1769078131.823952:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078131.823962:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078131.824003:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961489536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078131.824035:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078131.824038:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078131.824059:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078131.824062:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3814 00000100:00080000:0.0:1769078131.824067:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078131.824071:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078131.824083:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078131.824098:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961489664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078131.824110:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078131.824114:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078131.824117:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078131.824121:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3814 00000100:00080000:0.0:1769078131.824125:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078131.824130:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961489536:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078131.824133:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078131.824143:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078131.824156:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961489792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078131.824257:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961489664:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078131.824295:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961489792:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078132.335845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078132.527914:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078133.359811:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078133.551901:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078134.383913:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078134.575950:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769078134.896018:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769078134.896063:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078134.896081:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078134.896126:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078134.896137:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769078134.896229:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078131/real 1769078134] req@ffff895e4383c000 x1855008961489792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078134.896255:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961489792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078134.896268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078134.896273:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078134.896278:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961489792:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078134.896287:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078131/real 1769078134] req@ffff895e4383f480 x1855008961489536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078134.896321:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961489536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078134.896329:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078134.896331:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078134.896334:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961489536:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078134.896340:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078131/real 1769078134] req@ffff895e4383ce00 x1855008961489664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078134.896353:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961489664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078134.896361:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078134.896373:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078134.896377:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961489664:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078135.407855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078135.599933:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078136.431887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078136.623917:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078136.943897:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078136.943908:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078136.943913:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078136.943922:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3819 00000100:00080000:2.0:1769078136.943932:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078136.943939:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078136.943975:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961489920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078136.944010:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078136.944014:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078136.944016:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078136.944021:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3819 00000100:00080000:2.0:1769078136.944025:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078136.944041:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078136.944051:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078136.944065:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961490048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078136.944097:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078136.944101:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078136.944102:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078136.944106:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3819 00000100:00080000:2.0:1769078136.944112:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078136.944116:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078136.944122:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078136.944136:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961490176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078136.944152:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961489920:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078136.944224:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078136.944265:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078136/real 1769078136] req@ffff895e44e2bb80 x1855008961489920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078136.944284:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961489920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078136.944295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078136.944300:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078136.944303:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961489920:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078136.944319:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961490048:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078136.944343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078136.944389:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078136/real 1769078136] req@ffff895e44e2b800 x1855008961490048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078136.944402:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961490048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078136.944410:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078136.944413:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078136.944417:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961490048:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078136.944449:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961490176:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078136.944471:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078136.944482:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078136/real 1769078136] req@ffff895e44e29c00 x1855008961490176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078136.944493:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961490176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078136.944500:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078136.944503:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078136.944506:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961490176:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078137.455883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078137.647875:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078138.479846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078138.671929:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078139.503823:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078139.695902:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078140.527848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078140.719904:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078141.551853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078141.743914:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078142.063883:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078142.063893:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078142.063897:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078142.063908:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3824 00000100:00080000:1.0:1769078142.063918:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078142.063925:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078142.063961:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961490304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078142.064011:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078142.064015:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078142.064017:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078142.064021:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3824 00000100:00080000:1.0:1769078142.064026:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078142.064039:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078142.064049:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078142.064060:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961490304:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078142.064066:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961490432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078142.064080:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078142.064083:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078142.064085:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078142.064090:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3824 00000100:00080000:1.0:1769078142.064095:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078142.064100:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078142.064104:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078142.064115:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961490560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078142.064122:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078142.064160:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078142/real 1769078142] req@ffff895e4713dc00 x1855008961490304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078142.064174:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961490304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078142.064182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078142.064184:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078142.064187:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961490304:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078142.064206:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961490432:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078142.064220:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078142.064228:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078142/real 1769078142] req@ffff895e4713f800 x1855008961490432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078142.064235:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961490432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078142.064241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078142.064243:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078142.064245:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961490432:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078142.064249:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961490560:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078142.064258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078142.064290:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078142/real 1769078142] req@ffff895e4713ca80 x1855008961490560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078142.064309:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961490560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078142.064314:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078142.064317:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078142.064319:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961490560:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078142.575851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078142.767910:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078143.599835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078143.791895:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078144.623870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078144.815907:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078145.647872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078145.839880:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078146.671845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078146.863922:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078147.183890:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078147.183899:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078147.183903:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078147.183911:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3829 00000100:00080000:2.0:1769078147.183923:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078147.183930:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078147.183970:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961490688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078147.184018:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078147.184021:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078147.184023:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078147.184026:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3829 00000100:00080000:2.0:1769078147.184029:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078147.184032:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078147.184040:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078147.184052:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961490816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078147.184059:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078147.184062:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078147.184063:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078147.184066:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3829 00000100:00080000:2.0:1769078147.184070:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078147.184071:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961490688:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078147.184075:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078147.184081:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078147.184089:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961490944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078147.184134:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078147.184173:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078147/real 1769078147] req@ffff895e44e29c00 x1855008961490688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078147.184214:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961490688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078147.184222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078147.184225:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078147.184228:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961490688:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078147.184248:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961490816:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078147.184267:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078147.184276:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078147/real 1769078147] req@ffff895e44e2b800 x1855008961490816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078147.184292:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961490816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078147.184298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078147.184300:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078147.184303:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961490816:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078147.184307:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961490944:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078147.184320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078147.184326:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078147/real 1769078147] req@ffff895e44e2bb80 x1855008961490944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078147.184334:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961490944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078147.184339:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078147.184341:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078147.184342:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961490944:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078147.695820:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078147.887911:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078148.719863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078148.911935:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078149.743851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078149.935932:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078150.767823:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078150.959925:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078151.791820:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078151.983914:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078152.303871:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078152.303897:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078152.303901:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078152.303909:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3834 00000100:00080000:2.0:1769078152.303919:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078152.303925:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078152.303954:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961491072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078152.303977:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078152.303980:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078152.303981:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078152.303984:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3834 00000100:00080000:2.0:1769078152.303986:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078152.303989:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078152.303995:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078152.304004:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961491200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078152.304012:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078152.304014:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078152.304016:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078152.304020:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3834 00000100:00100000:3.0:1769078152.304021:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961491072:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078152.304022:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078152.304029:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078152.304033:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078152.304044:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961491328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078152.304083:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078152.304148:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078152/real 1769078152] req@ffff895e44e29180 x1855008961491072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078152.304170:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961491072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078152.304177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078152.304180:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078152.304183:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961491072:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078152.304204:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961491200:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078152.304221:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078152.304230:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078152/real 1769078152] req@ffff895e44e28a80 x1855008961491200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078152.304237:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961491200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078152.304243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078152.304245:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078152.304247:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961491200:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078152.304251:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961491328:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078152.304260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078152.304266:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078152/real 1769078152] req@ffff895e44e29500 x1855008961491328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078152.304273:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961491328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078152.304287:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078152.304290:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078152.304292:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961491328:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078152.815867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078153.007900:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078153.839895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078154.031940:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078154.863878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078155.055982:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078155.887905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078156.079946:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078156.911871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078157.103910:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078157.423898:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078157.423909:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078157.423913:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078157.423923:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3839 00000100:00080000:2.0:1769078157.423936:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078157.423943:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078157.423985:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961491456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078157.424039:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078157.424042:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078157.424044:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078157.424048:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3839 00000100:00080000:2.0:1769078157.424051:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078157.424054:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078157.424063:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078157.424074:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961491584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078157.424083:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078157.424085:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078157.424087:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078157.424109:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3839 00000100:00080000:2.0:1769078157.424112:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078157.424118:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078157.424123:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078157.424130:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961491456:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078157.424134:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961491712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078157.424217:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078157.424263:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078157/real 1769078157] req@ffff895e44e29f80 x1855008961491456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078157.424283:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961491456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078157.424293:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078157.424298:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078157.424303:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961491456:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078157.424310:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961491584:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078157.424333:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078157.424346:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078157/real 1769078157] req@ffff895e44e28e00 x1855008961491584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078157.424358:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961491584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078157.424379:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078157.424384:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078157.424387:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961491584:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078157.424414:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961491712:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078157.424436:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078157.424447:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078157/real 1769078157] req@ffff895e44e2b480 x1855008961491712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078157.424458:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961491712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078157.424466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078157.424469:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078157.424472:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961491712:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078157.935830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078158.127903:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078158.959851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078159.151933:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078159.983889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078160.176003:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078161.007906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078161.199903:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078162.031858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078162.223938:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078162.543849:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078162.543857:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078162.543860:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078162.543867:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3845 00000100:00080000:2.0:1769078162.543877:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078162.543882:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078162.543913:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961491840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078162.543946:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078162.543949:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078162.543950:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078162.543953:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3845 00000100:00080000:2.0:1769078162.543955:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078162.543958:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078162.543964:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078162.543985:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961491968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078162.543992:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078162.543994:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078162.543996:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078162.543999:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3845 00000100:00080000:2.0:1769078162.544002:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078162.544005:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078162.544008:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078162.544016:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961492096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078162.544036:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961491840:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078162.544117:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078162.544178:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078162/real 1769078162] req@ffff895e44e2aa00 x1855008961491840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078162.544201:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961491840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078162.544214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078162.544220:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078162.544235:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961491840:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078162.544242:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961491968:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078162.544263:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078162.544274:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078162/real 1769078162] req@ffff895e44e2b100 x1855008961491968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078162.544283:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961491968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078162.544289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078162.544292:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078162.544294:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961491968:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078162.544321:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961492096:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078162.544332:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078162.544339:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078162/real 1769078162] req@ffff895e44e2a300 x1855008961492096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078162.544346:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961492096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078162.544352:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078162.544354:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078162.544356:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961492096:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078163.055870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078163.247949:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078164.079889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078164.272002:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078165.103868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078165.295853:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078166.127887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078166.319930:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078167.151871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078167.343949:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078167.663886:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078167.663915:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078167.663920:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078167.663930:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3850 00000100:00080000:0.0:1769078167.663941:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078167.663949:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078167.663981:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961492224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078167.664003:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078167.664005:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078167.664007:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078167.664010:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3850 00000100:00080000:0.0:1769078167.664012:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078167.664016:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078167.664023:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078167.664032:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961492352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078167.664047:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078167.664049:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078167.664051:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078167.664057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3850 00000100:00080000:0.0:1769078167.664062:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078167.664063:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961492224:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078167.664067:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078167.664073:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078167.664082:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961492480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078167.664124:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078167.664165:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078167/real 1769078167] req@ffff895e4383c380 x1855008961492224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078167.664179:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961492224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078167.664186:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078167.664190:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078167.664192:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961492224:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078167.664198:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961492352:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078167.664213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078167.664220:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078167/real 1769078167] req@ffff895e4383f100 x1855008961492352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078167.664228:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961492352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078167.664234:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078167.664236:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078167.664238:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961492352:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078167.664256:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961492480:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078167.664266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078167.664272:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078167/real 1769078167] req@ffff895e4383fb80 x1855008961492480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078167.664279:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961492480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078167.664293:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078167.664295:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078167.664298:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961492480:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078168.175885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078168.367963:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078169.199871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078169.391951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078170.223903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078170.415986:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078171.247923:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078171.439973:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078172.271916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078172.463925:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078172.783907:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078172.783918:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078172.783922:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078172.783930:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3855 00000100:00080000:3.0:1769078172.783940:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078172.783946:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078172.783977:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961492608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078172.784001:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078172.784004:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078172.784006:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078172.784008:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3855 00000100:00080000:3.0:1769078172.784011:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078172.784013:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078172.784020:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078172.784029:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961492736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078172.784037:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078172.784040:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769078172.784048:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961492608:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078172.784059:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078172.784065:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3855 00000100:00080000:3.0:1769078172.784079:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078172.784084:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078172.784091:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078172.784105:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961492864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078172.784116:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078172.784150:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078172/real 1769078172] req@ffff895e453d5500 x1855008961492608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078172.784164:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961492608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078172.784172:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078172.784175:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078172.784178:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961492608:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078172.784196:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961492736:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078172.784211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078172.784219:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078172/real 1769078172] req@ffff895e453d4a80 x1855008961492736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078172.784227:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961492736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078172.784233:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078172.784235:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078172.784238:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961492736:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078172.784241:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961492864:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078172.784261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078172.784267:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078172/real 1769078172] req@ffff895e453d7480 x1855008961492864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078172.784275:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961492864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078172.784281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078172.784283:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078172.784285:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961492864:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078173.295905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078173.487920:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078174.319902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078174.511971:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078175.343849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078175.535981:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078176.367870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078176.559919:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078177.391897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078177.583911:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078177.903832:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078177.903840:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078177.903844:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078177.903852:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3860 00000100:00080000:3.0:1769078177.903861:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078177.903867:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078177.903899:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961492992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078177.903923:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078177.903925:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078177.903926:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078177.903929:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3860 00000100:00080000:3.0:1769078177.903931:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078177.903933:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078177.903940:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078177.903948:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961493120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078177.903973:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078177.903977:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961492992:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078177.903977:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078177.903982:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078177.903991:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3860 00000100:00080000:3.0:1769078177.903995:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078177.904000:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078177.904005:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078177.904019:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961493248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078177.904051:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078177.904092:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078177/real 1769078177] req@ffff895e453d4e00 x1855008961492992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078177.904110:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961492992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078177.904118:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078177.904121:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078177.904123:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961492992:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078177.904155:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961493120:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078177.904179:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078177.904192:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078177/real 1769078177] req@ffff895e453d5f80 x1855008961493120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078177.904214:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961493120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078177.904224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078177.904227:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078177.904231:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961493120:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078177.904237:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961493248:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078177.904261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078177.904274:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078177/real 1769078177] req@ffff895e453d5c00 x1855008961493248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078177.904287:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961493248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078177.904295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078177.904298:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078177.904302:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961493248:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078178.415924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078178.607929:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078179.439909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078179.631924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078180.463906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078180.655882:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078181.487912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078181.679946:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078182.511884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078182.703918:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078183.023911:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078183.023925:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078183.023930:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078183.023943:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3865 00000100:00080000:2.0:1769078183.023957:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078183.023965:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078183.024013:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961493376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078183.024046:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078183.024050:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078183.024052:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078183.024056:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3865 00000100:00080000:2.0:1769078183.024060:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078183.024064:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078183.024076:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078183.024108:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961493504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078183.024122:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078183.024123:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961493376:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078183.024129:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078183.024134:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078183.024142:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3865 00000100:00080000:2.0:1769078183.024146:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078183.024152:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078183.024158:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078183.024173:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961493632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078183.024207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078183.024249:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078183/real 1769078183] req@ffff895e44e28000 x1855008961493376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078183.024267:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961493376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078183.024289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078183.024293:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078183.024295:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961493376:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078183.024319:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961493504:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078183.024336:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078183.024344:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078183/real 1769078183] req@ffff895e44e28380 x1855008961493504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078183.024352:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961493504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078183.024358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078183.024360:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078183.024362:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961493504:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078183.024365:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961493632:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078183.024374:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078183.024380:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078183/real 1769078183] req@ffff895e44e29880 x1855008961493632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078183.024386:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961493632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078183.024390:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078183.024392:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078183.024393:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961493632:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078183.535906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078183.727954:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078184.559878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078184.751889:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078185.583843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078185.775904:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078186.607899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078186.799943:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078187.631847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078187.823889:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078188.143865:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078188.143875:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078188.143879:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078188.143888:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3870 00000100:00080000:2.0:1769078188.143897:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078188.143902:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078188.143935:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961493760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078188.143973:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078188.143976:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078188.143977:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078188.143979:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3870 00000100:00080000:2.0:1769078188.143982:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078188.143985:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078188.143991:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078188.143999:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961493888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078188.144005:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078188.144007:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078188.144009:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078188.144011:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3870 00000100:00080000:2.0:1769078188.144012:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078188.144015:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078188.144018:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078188.144024:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961493760:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078188.144025:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961494016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078188.144084:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078188.144117:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078188/real 1769078188] req@ffff895e44e29880 x1855008961493760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078188.144130:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961493760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078188.144136:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078188.144140:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078188.144147:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961493760:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078188.144152:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961493888:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078188.144164:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078188.144172:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078188/real 1769078188] req@ffff895e44e28380 x1855008961493888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078188.144181:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961493888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078188.144188:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078188.144191:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078188.144194:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961493888:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078188.144228:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961494016:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078188.144328:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078188.144338:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078188/real 1769078188] req@ffff895e44e28000 x1855008961494016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078188.144346:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961494016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078188.144351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078188.144353:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078188.144357:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961494016:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078188.655855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078188.847902:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078189.679849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078189.871879:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078190.703863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078190.895915:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078191.727856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078191.919896:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078192.751822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078192.943920:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078193.263865:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078193.263872:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078193.263876:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078193.263883:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3875 00000100:00080000:3.0:1769078193.263892:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078193.263898:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078193.263927:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961494144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078193.263947:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078193.263949:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078193.263962:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078193.263965:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3875 00000100:00080000:3.0:1769078193.263969:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078193.263971:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078193.263978:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078193.263985:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961494144:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078193.263990:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961494272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078193.263998:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078193.264000:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078193.264001:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078193.264004:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3875 00000100:00080000:3.0:1769078193.264006:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078193.264008:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078193.264011:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078193.264017:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961494400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078193.264049:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078193.264083:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078193/real 1769078193] req@ffff895e453d7800 x1855008961494144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078193.264107:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961494144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078193.264114:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078193.264117:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078193.264119:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961494144:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078193.264138:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961494272:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078193.264169:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078193.264182:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078193/real 1769078193] req@ffff895e453d5c00 x1855008961494272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078193.264193:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961494272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078193.264200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078193.264203:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078193.264206:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961494272:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078193.264212:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961494400:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078193.264229:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078193.264238:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078193/real 1769078193] req@ffff895e453d5f80 x1855008961494400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078193.264247:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961494400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078193.264253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078193.264256:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078193.264258:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961494400:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078193.775829:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078193.967892:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078194.799850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078194.991911:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078195.823838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078196.015885:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078196.847883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078197.039888:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078197.871830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078198.063868:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078198.383846:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078198.383856:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078198.383861:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078198.383870:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3880 00000100:00080000:3.0:1769078198.383898:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078198.383905:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078198.383938:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961494528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078198.383968:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078198.383972:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078198.383974:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078198.383978:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3880 00000100:00080000:3.0:1769078198.383981:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078198.383985:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078198.383994:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078198.384007:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961494656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078198.384012:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961494528:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078198.384025:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078198.384029:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078198.384031:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078198.384035:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3880 00000100:00080000:3.0:1769078198.384039:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078198.384043:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078198.384049:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078198.384060:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961494784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078198.384110:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961494656:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078198.384145:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961494784:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078198.895839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078199.087908:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078199.919872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078200.111862:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078200.943851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078201.135915:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769078201.456008:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769078201.456048:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769078201.456064:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769078201.456106:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769078201.456113:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769078201.456157:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078198/real 1769078201] req@ffff895e453d7480 x1855008961494656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078201.456185:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961494656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078201.456199:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078201.456205:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078201.456208:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961494656:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078201.456217:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078198/real 1769078201] req@ffff895e453d4a80 x1855008961494784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078201.456228:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961494784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078201.456235:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078201.456239:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078201.456242:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961494784:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078201.456248:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078198/real 1769078201] req@ffff895e453d4e00 x1855008961494528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078201.456260:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961494528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078201.456297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078201.456301:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078201.456321:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961494528:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078201.967846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078202.159864:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078202.991845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078203.183908:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078203.503864:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078203.503875:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078203.503879:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078203.503896:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3886 00000100:00080000:0.0:1769078203.503904:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078203.503910:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078203.503942:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961494912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078203.503973:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078203.503976:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078203.503978:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078203.503981:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3886 00000100:00080000:0.0:1769078203.503985:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078203.503989:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078203.504001:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078203.504019:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961494912:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078203.504020:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961495040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078203.504036:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078203.504039:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078203.504041:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078203.504044:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3886 00000100:00080000:0.0:1769078203.504047:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078203.504067:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078203.504073:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078203.504086:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961495168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078203.504108:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078203.504155:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078203/real 1769078203] req@ffff895e4383c700 x1855008961494912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078203.504173:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961494912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078203.504181:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078203.504186:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078203.504189:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961494912:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078203.504216:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961495040:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078203.504265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078203.504281:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078203/real 1769078203] req@ffff895e4383e680 x1855008961495040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078203.504292:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961495040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078203.504299:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078203.504302:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078203.504305:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961495040:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078203.504311:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961495168:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078203.504340:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078203.504352:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078203/real 1769078203] req@ffff895e4383d880 x1855008961495168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078203.504361:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961495168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078203.504369:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078203.504372:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078203.504374:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961495168:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078204.015854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078204.207869:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078205.039839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078205.231892:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078206.063829:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078206.255897:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078207.087838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078207.279893:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078208.111815:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078208.303884:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078208.623857:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078208.623867:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078208.623872:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078208.623880:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3891 00000100:00080000:1.0:1769078208.623889:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078208.623896:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078208.623940:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961495296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078208.623964:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078208.623966:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078208.623967:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078208.623970:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3891 00000100:00080000:1.0:1769078208.623972:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078208.623975:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078208.623982:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078208.623991:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961495424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078208.624003:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961495296:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769078208.624015:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078208.624018:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078208.624019:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078208.624023:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3891 00000100:00080000:1.0:1769078208.624027:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078208.624031:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078208.624036:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078208.624047:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961495552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078208.624055:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078208.624094:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078208/real 1769078208] req@ffff895e4713f480 x1855008961495296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078208.624108:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961495296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078208.624118:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078208.624122:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078208.624125:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961495296:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078208.624170:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961495424:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078208.624200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078208.624209:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078208/real 1769078208] req@ffff895e4713ce00 x1855008961495424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078208.624217:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961495424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078208.624222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078208.624224:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078208.624227:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961495424:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078208.624230:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961495552:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078208.624241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078208.624246:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078208/real 1769078208] req@ffff895e4713d500 x1855008961495552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078208.624253:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961495552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078208.624260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078208.624263:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078208.624266:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961495552:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078209.135850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078209.327906:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078210.159884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078210.351909:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078211.183846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078211.375885:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078212.207860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078212.399888:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078213.231847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078213.423919:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078213.743872:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078213.743880:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078213.743883:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078213.743890:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3896 00000100:00080000:0.0:1769078213.743899:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078213.743905:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078213.743935:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961495680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078213.743955:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078213.743958:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078213.743960:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078213.743963:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3896 00000100:00080000:0.0:1769078213.743967:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078213.743969:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078213.743978:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078213.743988:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961495808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078213.743993:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961495680:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078213.744018:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078213.744021:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078213.744023:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078213.744026:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3896 00000100:00080000:0.0:1769078213.744029:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078213.744032:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078213.744036:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078213.744045:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961495936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078213.744047:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078213.744110:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078213/real 1769078213] req@ffff895e4383d180 x1855008961495680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078213.744131:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961495680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078213.744137:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078213.744141:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078213.744143:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961495680:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078213.744162:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961495808:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078213.744179:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078213.744194:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078213/real 1769078213] req@ffff895e4383df80 x1855008961495808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078213.744201:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961495808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078213.744206:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078213.744208:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078213.744210:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961495808:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078213.744220:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961495936:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078213.744234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078213.744240:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078213/real 1769078213] req@ffff895e4383ed80 x1855008961495936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078213.744247:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961495936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078213.744259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078213.744261:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078213.744263:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961495936:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078214.255927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078214.447895:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078215.279850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078215.471881:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078216.303928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078216.495959:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078217.327870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078217.519930:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078218.351819:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078218.543944:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078218.863852:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078218.863867:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078218.863872:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078218.863883:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3901 00000100:00080000:0.0:1769078218.863897:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078218.863905:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078218.863950:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961496064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078218.864010:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078218.864014:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078218.864035:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078218.864039:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3901 00000100:00080000:0.0:1769078218.864044:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078218.864049:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078218.864057:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961496064:192.168.204.117@tcp:250:kworker.0 00000008:00100000:0.0:1769078218.864059:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078218.864078:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961496192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078218.864090:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078218.864094:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078218.864096:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078218.864099:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3901 00000100:00080000:0.0:1769078218.864104:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078218.864108:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078218.864113:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078218.864125:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961496320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078218.864130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078218.864169:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078218/real 1769078218] req@ffff895e4383ed80 x1855008961496064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078218.864186:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961496064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078218.864196:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078218.864200:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078218.864204:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961496064:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078218.864230:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961496192:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078218.864250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078218.864259:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078218/real 1769078218] req@ffff895e4383df80 x1855008961496192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078218.864266:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961496192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078218.864272:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078218.864274:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078218.864276:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961496192:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078218.864288:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961496320:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078218.864301:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078218.864307:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078218/real 1769078218] req@ffff895e4383d180 x1855008961496320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078218.864314:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961496320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078218.864319:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078218.864321:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078218.864323:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961496320:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078219.375907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078219.567915:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078220.399843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078220.591916:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078221.423849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078221.615861:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078222.447888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078222.639964:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078223.471850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078223.663920:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078223.983908:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078223.983919:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078223.983923:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078223.983933:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3906 00000100:00080000:1.0:1769078223.983943:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078223.983950:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078223.983985:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961496448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078223.984016:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078223.984019:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078223.984021:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078223.984025:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3906 00000100:00080000:1.0:1769078223.984029:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078223.984032:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078223.984055:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078223.984060:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961496448:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078223.984071:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961496576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078223.984082:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078223.984085:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078223.984087:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078223.984091:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3906 00000100:00080000:1.0:1769078223.984095:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078223.984099:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078223.984104:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769078223.984111:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078223.984116:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961496704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078223.984144:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078223/real 1769078223] req@ffff895e4713ea00 x1855008961496448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078223.984158:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961496448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078223.984165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078223.984167:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078223.984170:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961496448:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078223.984187:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961496576:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078223.984200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078223.984216:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078223/real 1769078223] req@ffff895e4713f100 x1855008961496576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078223.984230:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961496576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078223.984235:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078223.984237:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078223.984239:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961496576:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078223.984243:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961496704:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078223.984256:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078223.984262:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078223/real 1769078223] req@ffff895e4713e300 x1855008961496704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078223.984269:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961496704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078223.984273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078223.984274:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078223.984276:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961496704:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078224.495849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078224.687913:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078225.519813:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078225.711919:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078226.543862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078226.735912:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078227.567911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078227.759901:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078228.591856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078228.783896:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078229.103885:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078229.103895:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078229.103899:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078229.103906:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3911 00000100:00080000:2.0:1769078229.103915:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078229.103935:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078229.103967:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961496832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078229.103993:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078229.103997:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078229.103999:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078229.104003:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3911 00000100:00080000:2.0:1769078229.104007:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078229.104010:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078229.104020:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078229.104032:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961496960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078229.104033:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961496832:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078229.104049:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078229.104052:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078229.104054:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078229.104058:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3911 00000100:00080000:2.0:1769078229.104061:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078229.104065:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078229.104069:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078229.104083:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961497088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078229.104105:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078229.104142:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078229/real 1769078229] req@ffff895e44e2a300 x1855008961496832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078229.104159:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961496832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078229.104180:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078229.104184:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078229.104188:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961496832:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078229.104209:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961496960:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078229.104227:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078229.104238:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078229/real 1769078229] req@ffff895e44e2b100 x1855008961496960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078229.104249:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961496960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078229.104257:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078229.104261:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078229.104264:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961496960:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078229.104270:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961497088:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078229.104287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078229.104297:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078229/real 1769078229] req@ffff895e44e2aa00 x1855008961497088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078229.104304:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961497088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078229.104308:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078229.104310:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078229.104312:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961497088:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078229.615866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078229.807933:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078230.639910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078230.831917:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078231.663850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078231.855905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078232.687857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078232.879909:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078233.711855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078233.903906:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078234.223858:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078234.223866:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078234.223869:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078234.223877:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3916 00000100:00080000:2.0:1769078234.223886:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078234.223892:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078234.223922:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961497216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078234.223945:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078234.223947:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078234.223948:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078234.223951:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3916 00000100:00080000:2.0:1769078234.223953:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078234.223956:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078234.223962:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078234.223969:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961497344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078234.223976:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078234.223978:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078234.223980:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078234.223982:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3916 00000100:00080000:2.0:1769078234.223984:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078234.223986:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078234.223988:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078234.223993:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961497216:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078234.224009:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961497472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078234.224047:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078234.224076:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078234/real 1769078234] req@ffff895e44e2b480 x1855008961497216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078234.224088:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961497216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078234.224095:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078234.224098:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078234.224106:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961497216:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078234.224111:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961497344:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078234.224123:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078234.224130:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078234/real 1769078234] req@ffff895e44e28e00 x1855008961497344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078234.224137:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961497344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078234.224142:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078234.224144:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078234.224146:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961497344:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078234.224165:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961497472:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078234.224174:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078234.224179:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078234/real 1769078234] req@ffff895e44e29f80 x1855008961497472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078234.224191:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961497472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078234.224196:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078234.224198:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078234.224200:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961497472:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078234.735853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078234.927909:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078235.759828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078235.952822:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078236.783835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078236.975877:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078237.807839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078237.999908:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078238.831859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078239.023889:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078239.343859:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078239.343866:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078239.343870:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078239.343877:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3921 00000100:00080000:1.0:1769078239.343886:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078239.343891:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078239.343920:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961497600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078239.343940:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078239.343942:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078239.343950:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078239.343953:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3921 00000100:00080000:1.0:1769078239.343978:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078239.343984:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078239.343988:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961497600:192.168.204.117@tcp:250:kworker.0 00000008:00100000:1.0:1769078239.343994:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078239.344011:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961497728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078239.344020:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078239.344022:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078239.344023:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078239.344026:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3921 00000100:00080000:1.0:1769078239.344029:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078239.344031:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078239.344034:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078239.344042:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961497856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078239.344048:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078239.344086:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078239/real 1769078239] req@ffff895e4713d880 x1855008961497600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078239.344103:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961497600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078239.344112:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078239.344116:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078239.344119:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961497600:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078239.344140:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961497728:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078239.344158:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078239.344168:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078239/real 1769078239] req@ffff895e4713ed80 x1855008961497728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078239.344178:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961497728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078239.344202:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078239.344206:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078239.344209:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961497728:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078239.344214:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961497856:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078239.344232:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078239.344241:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078239/real 1769078239] req@ffff895e4713c000 x1855008961497856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078239.344251:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961497856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078239.344258:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078239.344261:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078239.344264:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961497856:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078239.855839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078240.047913:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078240.879830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078241.071905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078241.903850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078242.095893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078242.927839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078243.119869:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078243.951954:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078244.143889:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078244.463858:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078244.463869:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078244.463872:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078244.463881:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3927 00000100:00080000:1.0:1769078244.463891:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078244.463898:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078244.463930:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961497984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078244.463953:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078244.463968:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078244.463970:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078244.463973:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3927 00000100:00080000:1.0:1769078244.463977:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078244.463982:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078244.463991:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078244.464001:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961497984:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078244.464007:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961498112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078244.464019:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078244.464022:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078244.464024:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078244.464028:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3927 00000100:00080000:1.0:1769078244.464032:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078244.464035:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078244.464040:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078244.464050:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961498240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078244.464063:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078244.464094:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078244/real 1769078244] req@ffff895e4713c700 x1855008961497984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078244.464106:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961497984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078244.464113:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078244.464117:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078244.464127:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961497984:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078244.464149:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961498112:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078244.464171:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078244.464184:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078244/real 1769078244] req@ffff895e4713fb80 x1855008961498112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078244.464196:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961498112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078244.464203:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078244.464206:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078244.464209:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961498112:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078244.464213:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961498240:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078244.464237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078244.464247:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078244/real 1769078244] req@ffff895e4713c000 x1855008961498240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078244.464258:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961498240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078244.464265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078244.464268:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078244.464271:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961498240:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078244.975843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078245.167922:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078245.999850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078246.191904:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078247.023845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078247.215926:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078248.047868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078248.239933:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078249.071828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078249.263908:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078249.583862:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078249.583870:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078249.583875:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078249.583882:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3932 00000100:00080000:2.0:1769078249.583892:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078249.583899:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078249.583933:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961498368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078249.583960:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078249.583963:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078249.583965:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078249.583968:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3932 00000100:00080000:2.0:1769078249.583972:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078249.583976:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078249.583996:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961498368:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769078249.584011:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078249.584028:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961498496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078249.584039:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078249.584042:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078249.584044:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078249.584048:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3932 00000100:00080000:2.0:1769078249.584052:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078249.584057:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:3.0:1769078249.584063:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:2.0:1769078249.584064:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078249.584076:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961498624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078249.584111:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078249/real 1769078249] req@ffff895e44e29500 x1855008961498368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078249.584128:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961498368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078249.584137:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078249.584142:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078249.584145:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961498368:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078249.584167:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961498496:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078249.584186:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078249.584207:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078249/real 1769078249] req@ffff895e44e28a80 x1855008961498496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078249.584218:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961498496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078249.584226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078249.584229:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078249.584239:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961498496:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078249.584245:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961498624:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078249.584261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078249.584271:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078249/real 1769078249] req@ffff895e44e29180 x1855008961498624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078249.584280:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961498624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078249.584297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078249.584300:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078249.584304:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961498624:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078250.096003:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078250.287889:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078251.119873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078251.311893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078252.143859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078252.335914:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078253.167916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078253.359912:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078254.191850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078254.383893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078254.703860:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078254.703870:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078254.703875:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078254.703885:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3937 00000100:00080000:2.0:1769078254.703895:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078254.703901:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078254.703932:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961498752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078254.703974:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078254.703977:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078254.703979:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078254.703983:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3937 00000100:00080000:2.0:1769078254.703987:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078254.703992:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078254.704001:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078254.704012:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961498880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078254.704019:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961498752:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078254.704020:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078254.704027:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078254.704029:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078254.704032:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3937 00000100:00080000:2.0:1769078254.704041:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078254.704045:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078254.704050:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078254.704062:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961499008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078254.704082:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078254.704117:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078254/real 1769078254] req@ffff895e44e2bb80 x1855008961498752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078254.704134:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961498752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078254.704143:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078254.704146:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078254.704150:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961498752:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078254.704170:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961498880:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078254.704189:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078254.704199:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078254/real 1769078254] req@ffff895e44e2b800 x1855008961498880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078254.704209:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961498880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078254.704216:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078254.704220:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078254.704223:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961498880:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078254.704234:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961499008:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078254.704250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078254.704259:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078254/real 1769078254] req@ffff895e44e29c00 x1855008961499008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078254.704269:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961499008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078254.704275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078254.704278:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078254.704281:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961499008:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078255.215857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078255.407889:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078256.239858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078256.431896:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078257.263853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078257.455905:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078258.287846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078258.479892:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078259.311853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078259.503937:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078259.823853:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078259.823861:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078259.823864:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078259.823871:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3942 00000100:00080000:0.0:1769078259.823879:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078259.823884:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078259.823913:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961499136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078259.823933:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078259.823935:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078259.823936:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078259.823938:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3942 00000100:00080000:0.0:1769078259.823940:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078259.823943:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078259.823949:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078259.823957:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961499264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078259.823964:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961499136:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078259.823994:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078259.823997:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078259.823999:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078259.824004:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3942 00000100:00080000:0.0:1769078259.824008:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078259.824012:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078259.824017:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769078259.824019:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078259.824032:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961499392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078259.824052:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078259/real 1769078259] req@ffff895e4383d880 x1855008961499136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078259.824064:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961499136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078259.824071:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078259.824073:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078259.824075:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961499136:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078259.824091:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961499264:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078259.824102:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078259.824109:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078259/real 1769078259] req@ffff895e4383e680 x1855008961499264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078259.824122:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961499264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078259.824127:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078259.824129:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078259.824131:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961499264:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078259.824134:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961499392:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078259.824145:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078259.824150:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078259/real 1769078259] req@ffff895e4383c700 x1855008961499392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078259.824157:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961499392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078259.824161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078259.824163:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078259.824164:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961499392:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078260.335858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078260.527915:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078261.359935:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078261.551898:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078262.383952:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078262.575914:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078263.407878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078263.599992:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078264.431885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078264.623933:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078264.943837:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078264.943845:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078264.943848:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078264.943861:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3947 00000100:00080000:0.0:1769078264.943873:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078264.943880:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078264.943914:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961499520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078264.943952:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078264.943954:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078264.943955:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078264.943958:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3947 00000100:00080000:0.0:1769078264.943961:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078264.943964:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078264.943971:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078264.943980:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961499648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078264.943986:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078264.943988:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078264.943990:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078264.943991:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3947 00000100:00080000:0.0:1769078264.943993:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078264.943997:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078264.944000:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961499520:192.168.204.117@tcp:250:kworker.0 00000008:00100000:0.0:1769078264.944001:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078264.944009:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961499776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078264.944089:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961499648:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078264.944127:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961499776:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078265.455841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078265.647886:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078266.479884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078266.671961:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078267.503870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078267.695914:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769078268.016031:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769078268.016074:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769078268.016090:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769078268.016214:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078264/real 1769078268] req@ffff895e4383fb80 x1855008961499520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078289 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078268.016258:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961499520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078289 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078268.016269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078268.016274:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078268.016278:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961499520:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078268.016351:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769078268.016430:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078264/real 1769078268] req@ffff895e4383f100 x1855008961499648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078289 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078268.016444:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961499648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078289 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078268.016452:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078268.016456:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078268.016459:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961499648:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078268.016499:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769078268.016735:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078264/real 1769078268] req@ffff895e4383c380 x1855008961499776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078268.016785:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961499776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078268.016793:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078268.016797:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078268.016800:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961499776:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078268.527815:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078268.719910:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078269.551855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078269.743879:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078270.063854:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078270.063864:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078270.063869:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078270.063878:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3952 00000100:00080000:0.0:1769078270.063896:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078270.063905:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078270.063940:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961499904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078270.063970:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078270.063974:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078270.063976:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078270.063979:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3952 00000100:00080000:0.0:1769078270.063983:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078270.063986:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078270.063995:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078270.064008:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961500032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078270.064018:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769078270.064023:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961499904:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078270.064024:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078270.064026:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078270.064030:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3952 00000100:00080000:0.0:1769078270.064034:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078270.064048:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078270.064054:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078270.064066:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961500160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078270.064159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078270.064206:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078270/real 1769078270] req@ffff895e4383ce00 x1855008961499904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078270.064227:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961499904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078270.064237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078270.064242:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078270.064246:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961499904:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078270.064270:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961500032:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078270.064290:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078270.064303:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078270/real 1769078270] req@ffff895e4383f480 x1855008961500032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078270.064314:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961500032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078270.064323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078270.064326:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078270.064329:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961500032:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078270.064335:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961500160:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078270.064353:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078270.064363:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078270/real 1769078270] req@ffff895e4383c000 x1855008961500160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078270.064389:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961500160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078270.064397:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078270.064400:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078270.064417:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961500160:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078270.575871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078270.767878:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078271.599886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078271.791915:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078272.623876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078272.815940:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078273.647864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078273.839929:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078274.671844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078274.863905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078275.183861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078275.183871:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078275.183875:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078275.183884:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3957 00000100:00080000:0.0:1769078275.183897:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078275.183903:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078275.183939:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961500288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078275.183972:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078275.183975:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078275.183976:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078275.183979:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3957 00000100:00080000:0.0:1769078275.183982:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078275.183985:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078275.183992:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078275.184017:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961500416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078275.184038:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961500288:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078275.184060:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078275.184064:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078275.184066:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078275.184071:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3957 00000100:00080000:0.0:1769078275.184076:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078275.184080:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078275.184085:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769078275.184096:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078275.184099:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961500544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078275.184135:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078275/real 1769078275] req@ffff895e4383ea00 x1855008961500288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078275.184154:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961500288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078275.184164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078275.184168:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078275.184173:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961500288:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078275.184195:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961500416:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078275.184215:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078275.184224:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078275/real 1769078275] req@ffff895e4383dc00 x1855008961500416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078275.184248:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961500416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078275.184255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078275.184258:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078275.184262:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961500416:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078275.184267:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961500544:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078275.184283:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078275.184292:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078275/real 1769078275] req@ffff895e4383ca80 x1855008961500544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078275.184302:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961500544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078275.184309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078275.184312:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078275.184315:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961500544:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078275.695862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078275.887881:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078276.719844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078276.911935:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078277.743859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078277.936046:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078278.767796:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078278.960093:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078279.791856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078279.983852:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078280.303867:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078280.303875:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078280.303879:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078280.303907:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3962 00000100:00080000:3.0:1769078280.303927:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078280.303959:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078280.303992:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961500672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078280.304021:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078280.304023:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078280.304024:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078280.304027:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3962 00000100:00080000:3.0:1769078280.304030:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078280.304038:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078280.304045:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078280.304055:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961500800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078280.304062:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078280.304065:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769078280.304066:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961500672:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078280.304067:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078280.304075:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3962 00000100:00080000:3.0:1769078280.304078:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078280.304082:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078280.304087:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078280.304106:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961500928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078280.304126:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078280.304167:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078280/real 1769078280] req@ffff895e453d5500 x1855008961500672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078280.304185:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961500672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078280.304195:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078280.304199:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078280.304203:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961500672:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078280.304229:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961500800:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078280.304247:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078280.304257:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078280/real 1769078280] req@ffff895e453d7b80 x1855008961500800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078280.304267:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961500800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078280.304275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078280.304278:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078280.304281:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961500800:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078280.304286:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961500928:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078280.304300:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078280.304308:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078280/real 1769078280] req@ffff895e453d7100 x1855008961500928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078280.304318:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961500928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078280.304325:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078280.304328:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078280.304331:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961500928:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078280.815871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078281.007909:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078281.839834:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078282.031875:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078282.863854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078283.058199:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078283.887858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078284.079921:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078284.911849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078285.103901:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078285.423878:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078285.423889:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078285.423893:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078285.423903:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3967 00000100:00080000:3.0:1769078285.423914:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078285.423921:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078285.423954:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961501056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078285.423988:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078285.423992:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078285.423994:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078285.423998:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3967 00000100:00080000:3.0:1769078285.424002:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078285.424006:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078285.424015:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078285.424026:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961501056:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078285.424030:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961501184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078285.424041:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078285.424044:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078285.424045:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078285.424049:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3967 00000100:00080000:3.0:1769078285.424055:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078285.424061:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078285.424073:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078285.424083:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961501312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078285.424111:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078285.424143:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078285/real 1769078285] req@ffff895e453d4700 x1855008961501056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078285.424159:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961501056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078285.424168:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078285.424172:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078285.424175:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961501056:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078285.424197:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961501184:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078285.424216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078285.424224:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078285/real 1769078285] req@ffff895e453d5880 x1855008961501184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078285.424240:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961501184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078285.424245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078285.424247:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078285.424249:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961501184:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078285.424253:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961501312:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078285.424264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078285.424269:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078285/real 1769078285] req@ffff895e453d5180 x1855008961501312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078285.424285:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961501312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078285.424290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078285.424292:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078285.424294:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961501312:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078285.935831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078286.127951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078286.959864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078287.151905:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078287.983832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078288.175909:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078289.007871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078289.199904:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078290.031834:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078290.223941:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078290.543874:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078290.543900:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078290.543904:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078290.543914:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3973 00000100:00080000:1.0:1769078290.543933:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078290.543940:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078290.543973:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961501440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078290.543997:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078290.544001:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078290.544003:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078290.544006:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3973 00000100:00080000:1.0:1769078290.544009:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078290.544012:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078290.544021:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078290.544032:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961501568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078290.544039:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961501440:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769078290.544044:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078290.544049:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078290.544051:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078290.544055:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3973 00000100:00080000:1.0:1769078290.544066:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078290.544070:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078290.544075:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078290.544087:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961501696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078290.544092:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078290.544122:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078290/real 1769078290] req@ffff895e4713ed80 x1855008961501440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078290.544134:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961501440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078290.544142:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078290.544144:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078290.544146:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961501440:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078290.544196:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961501568:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078290.544213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078290.544221:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078290/real 1769078290] req@ffff895e4713d880 x1855008961501568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078290.544238:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961501568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078290.544244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078290.544246:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078290.544248:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961501568:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078290.544252:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961501696:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078290.544267:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078290.544274:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078290/real 1769078290] req@ffff895e4713e300 x1855008961501696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078290.544281:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961501696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078290.544286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078290.544288:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078290.544289:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961501696:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078291.055915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078291.247901:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078292.079890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078292.271953:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078293.103870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078293.295927:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078294.127928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078294.319883:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078295.151879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078295.343946:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078295.663913:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078295.663924:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078295.663928:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078295.663938:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3978 00000100:00080000:3.0:1769078295.663950:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078295.663956:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078295.664005:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961501824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078295.664034:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078295.664038:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078295.664040:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078295.664044:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3978 00000100:00080000:3.0:1769078295.664047:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078295.664051:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078295.664060:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078295.664071:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961501952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078295.664077:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078295.664080:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078295.664081:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078295.664084:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3978 00000100:00100000:0.0:1769078295.664099:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961501824:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078295.664105:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078295.664111:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078295.664114:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078295.664122:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961502080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078295.664186:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078295.664224:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078295/real 1769078295] req@ffff895e453d4380 x1855008961501824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078295.664246:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961501824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078295.664253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078295.664257:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078295.664260:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961501824:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078295.664265:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961501952:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078295.664282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078295.664289:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078295/real 1769078295] req@ffff895e453d6680 x1855008961501952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078295.664296:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961501952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078295.664302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078295.664304:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078295.664306:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961501952:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078295.664325:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961502080:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078295.664335:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078295.664340:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078295/real 1769078295] req@ffff895e453d6300 x1855008961502080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078295.664347:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961502080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078295.664360:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078295.664362:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078295.664364:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961502080:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078296.175877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078296.367960:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078297.199874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078297.391865:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078298.223850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078298.415902:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078299.247859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078299.439909:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078300.271865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078300.463875:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078300.783847:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078300.783861:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078300.783865:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078300.783874:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3983 00000100:00080000:3.0:1769078300.783884:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078300.783890:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078300.783935:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961502208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078300.783966:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078300.783969:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078300.783971:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078300.783975:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3983 00000100:00080000:3.0:1769078300.783979:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078300.783982:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078300.783991:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078300.784006:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961502336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078300.784015:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078300.784027:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961502208:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078300.784034:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078300.784038:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078300.784042:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3983 00000100:00080000:3.0:1769078300.784047:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078300.784051:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078300.784056:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078300.784067:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961502464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078300.784092:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078300.784132:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078300/real 1769078300] req@ffff895e453d6d80 x1855008961502208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078300.784150:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961502208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078300.784160:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078300.784164:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078300.784168:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961502208:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078300.784184:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961502336:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078300.784207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078300.784220:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078300/real 1769078300] req@ffff895e453d6300 x1855008961502336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078300.784238:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961502336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078300.784255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078300.784259:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078300.784262:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961502336:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078300.784284:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961502464:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078300.784304:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078300.784315:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078300/real 1769078300] req@ffff895e453d6680 x1855008961502464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078300.784326:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961502464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078300.784333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078300.784336:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078300.784339:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961502464:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078301.296845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078301.487915:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078302.319876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078302.511910:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078303.343836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078303.535897:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078304.367818:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078304.559883:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078305.391841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078305.583899:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078305.903897:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078305.903908:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078305.903912:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078305.903921:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3988 00000100:00080000:0.0:1769078305.903945:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078305.903951:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078305.903977:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961502592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078305.903999:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078305.904001:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078305.904002:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078305.904005:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3988 00000100:00080000:0.0:1769078305.904007:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078305.904009:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078305.904015:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078305.904022:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961502720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078305.904030:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078305.904032:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078305.904033:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078305.904038:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3988 00000100:00080000:0.0:1769078305.904041:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1769078305.904041:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961502592:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078305.904047:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078305.904054:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078305.904062:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961502848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078305.904119:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078305.904156:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078305/real 1769078305] req@ffff895e4383f800 x1855008961502592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078305.904182:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961502592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078305.904191:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078305.904196:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078305.904200:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961502592:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078305.904230:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961502720:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078305.904251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078305.904263:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078305/real 1769078305] req@ffff895e4383e300 x1855008961502720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078305.904274:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961502720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078305.904288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078305.904292:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078305.904296:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961502720:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078305.904301:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961502848:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078305.904323:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078305.904333:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078305/real 1769078305] req@ffff895e4383d500 x1855008961502848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078305.904343:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961502848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078305.904350:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078305.904354:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078305.904357:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961502848:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078306.415853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078306.607906:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078307.439815:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078307.631896:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078308.463852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078308.655896:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078309.487839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078309.679914:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078310.511865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078310.703884:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078311.027508:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078311.027527:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078311.027531:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078311.027540:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3993 00000100:00080000:2.0:1769078311.027550:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078311.027556:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078311.027589:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961502976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078311.027620:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078311.027623:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078311.027625:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078311.027629:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3993 00000100:00080000:2.0:1769078311.027632:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078311.027636:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078311.027644:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078311.027654:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961503104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078311.027674:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078311.027689:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078311.027691:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078311.027694:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3993 00000100:00080000:2.0:1769078311.027698:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078311.027701:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078311.027706:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078311.027715:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961503232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078311.028480:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961502976:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078311.028536:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078311.028568:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078311/real 1769078311] req@ffff895e44e29c00 x1855008961502976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078311.028585:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961502976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078311.028595:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078311.028599:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078311.028609:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961502976:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078311.028616:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961503104:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078311.028637:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078311.028648:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078311/real 1769078311] req@ffff895e44e2b800 x1855008961503104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078311.028660:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961503104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078311.028667:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078311.028671:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078311.028674:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961503104:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078311.028679:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961503232:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078311.028696:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078311.028706:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078311/real 1769078311] req@ffff895e44e2bb80 x1855008961503232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078311.028716:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961503232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078311.028723:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078311.028726:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078311.028729:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961503232:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078311.535904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078311.727946:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078312.559843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078312.751911:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078313.583879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078313.775881:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078314.607870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078314.799932:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078315.631911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078315.823898:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078316.143872:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078316.143882:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078316.143886:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078316.143893:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 3998 00000100:00080000:1.0:1769078316.143901:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078316.143906:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078316.143943:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961503360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078316.143964:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078316.143966:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078316.143968:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078316.143970:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3998 00000100:00080000:1.0:1769078316.143972:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078316.143974:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078316.143985:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078316.143993:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961503488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078316.144000:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078316.144003:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078316.144005:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078316.144009:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 3998 00000100:00100000:3.0:1769078316.144014:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961503360:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769078316.144015:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078316.144022:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078316.144050:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078316.144063:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961503616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078316.144075:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078316.144134:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078316/real 1769078316] req@ffff895e4713f100 x1855008961503360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078316.144152:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961503360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078316.144162:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078316.144167:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078316.144171:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961503360:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078316.144178:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961503488:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078316.144196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078316.144206:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078316/real 1769078316] req@ffff895e4713ea00 x1855008961503488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078316.144217:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961503488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078316.144224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078316.144227:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078316.144230:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961503488:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078316.144252:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961503616:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078316.144268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078316.144277:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078316/real 1769078316] req@ffff895e4713d500 x1855008961503616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078316.144287:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961503616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078316.144305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078316.144308:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078316.144311:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961503616:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078316.655871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078316.847959:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078317.679840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078317.871887:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078318.703894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078318.895875:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078319.727836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078319.919881:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078320.751836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078320.943875:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078321.263836:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078321.263844:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078321.263847:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078321.263855:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4003 00000100:00080000:1.0:1769078321.263862:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078321.263868:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078321.263897:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961503744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078321.263921:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078321.263923:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078321.263925:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078321.263927:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4003 00000100:00080000:1.0:1769078321.263929:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078321.263937:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078321.263944:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078321.263952:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961503872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078321.263959:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078321.263963:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078321.263964:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078321.263964:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961503744:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769078321.263971:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4003 00000100:00080000:1.0:1769078321.263987:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078321.263990:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078321.263993:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078321.264003:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961504000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078321.264019:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078321.264048:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078321/real 1769078321] req@ffff895e4713ce00 x1855008961503744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078321.264059:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961503744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078321.264065:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078321.264068:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078321.264071:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961503744:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078321.264087:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961503872:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078321.264099:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078321.264105:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078321/real 1769078321] req@ffff895e4713f480 x1855008961503872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078321.264112:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961503872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078321.264117:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078321.264119:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078321.264121:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961503872:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078321.264129:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961504000:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078321.264140:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078321.264146:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078321/real 1769078321] req@ffff895e4713ca80 x1855008961504000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078321.264152:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961504000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078321.264156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078321.264158:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078321.264160:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961504000:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078321.775867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078321.967920:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078322.799856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078322.991849:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078323.823847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078324.016654:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078324.847879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078325.040670:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078325.871889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078326.063979:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078326.383843:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078326.383851:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078326.383854:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078326.383861:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4008 00000100:00080000:1.0:1769078326.383870:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078326.383875:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078326.383903:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961504128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078326.383927:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078326.383929:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078326.383930:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078326.383933:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4008 00000100:00080000:1.0:1769078326.383935:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078326.383937:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078326.383942:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078326.383950:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961504256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078326.383956:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1769078326.383967:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961504128:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769078326.383974:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078326.383976:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078326.383979:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4008 00000100:00080000:1.0:1769078326.383981:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078326.383984:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078326.383987:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078326.383996:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961504384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078326.384031:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078326.384061:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078326/real 1769078326] req@ffff895e4713f800 x1855008961504128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078326.384073:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961504128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078326.384080:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078326.384083:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078326.384085:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961504128:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078326.384090:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961504256:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078326.384104:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078326.384110:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078326/real 1769078326] req@ffff895e4713dc00 x1855008961504256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078326.384123:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961504256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078326.384129:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078326.384130:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078326.384132:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961504256:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078326.384150:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961504384:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078326.384182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078326.384195:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078326/real 1769078326] req@ffff895e4713df80 x1855008961504384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078326.384203:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961504384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078326.384208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078326.384209:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078326.384211:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961504384:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078326.895828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078327.087964:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078327.919884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078328.111942:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078328.943831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078329.135946:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078329.967875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078330.159910:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078330.991871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078331.183906:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078331.503869:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078331.503879:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078331.503883:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078331.503892:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4014 00000100:00080000:0.0:1769078331.503901:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078331.503907:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078331.503938:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961504512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078331.503961:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078331.503963:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078331.503965:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078331.503967:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4014 00000100:00080000:0.0:1769078331.503969:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078331.503984:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078331.503992:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078331.504001:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961504640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078331.504005:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961504512:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078331.504011:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078331.504014:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078331.504015:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078331.504018:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4014 00000100:00080000:0.0:1769078331.504020:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078331.504023:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078331.504025:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078331.504032:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961504768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078331.504105:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961504640:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078331.504136:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961504768:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078332.015845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078332.207901:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078333.039855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078333.231922:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078334.063861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078334.255893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769078334.576013:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769078334.576079:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078334.576096:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078334.576140:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078334.576163:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769078334.576300:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078331/real 1769078334] req@ffff895e4383e300 x1855008961504640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078334.576351:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961504640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078334.576363:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078334.576368:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078334.576371:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961504640:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078334.576379:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078331/real 1769078334] req@ffff895e4383f800 x1855008961504768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078334.576390:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961504768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078334.576397:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078334.576400:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078334.576403:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961504768:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078334.576413:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078331/real 1769078334] req@ffff895e4383d500 x1855008961504512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078334.576425:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961504512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078334.576432:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078334.576435:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078334.576438:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961504512:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078335.087861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078335.279917:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078336.111854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078336.303953:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078336.623853:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078336.623863:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078336.623867:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078336.623877:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4019 00000100:00080000:0.0:1769078336.623887:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078336.623893:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078336.623926:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961504896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078336.623973:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078336.623976:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078336.623978:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078336.623983:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4019 00000100:00080000:0.0:1769078336.624018:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078336.624024:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078336.624034:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078336.624043:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961504896:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078336.624051:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961505024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078336.624065:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078336.624068:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078336.624071:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078336.624075:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4019 00000100:00080000:0.0:1769078336.624078:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078336.624082:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078336.624087:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078336.624098:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961505152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078336.624145:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078336.624196:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078336/real 1769078336] req@ffff895e4383ca80 x1855008961504896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078336.624216:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961504896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078336.624227:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078336.624232:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078336.624236:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961504896:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078336.624261:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961505024:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078336.624283:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078336.624296:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078336/real 1769078336] req@ffff895e4383dc00 x1855008961505024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078336.624308:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961505024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078336.624315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078336.624318:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078336.624321:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961505024:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078336.624327:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961505152:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078336.624344:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078336.624353:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078336/real 1769078336] req@ffff895e4383ea00 x1855008961505152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078336.624380:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961505152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078336.624387:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078336.624390:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078336.624393:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961505152:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078337.135858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078337.327888:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078338.159857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078338.351900:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078339.183887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078339.375923:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078340.207844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078340.399856:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078341.231852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078341.423903:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078341.743883:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078341.743892:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078341.743897:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078341.743917:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4024 00000100:00080000:2.0:1769078341.743943:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078341.743949:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078341.743979:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961505280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078341.744003:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078341.744006:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078341.744008:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078341.744012:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4024 00000100:00080000:2.0:1769078341.744015:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078341.744018:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078341.744026:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078341.744037:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961505408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078341.744047:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078341.744051:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078341.744054:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078341.744056:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961505280:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078341.744059:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4024 00000100:00080000:2.0:1769078341.744066:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078341.744069:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078341.744074:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078341.744084:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961505536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078341.744121:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078341.744156:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078341/real 1769078341] req@ffff895e44e29180 x1855008961505280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078341.744173:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961505280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078341.744183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078341.744187:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078341.744191:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961505280:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078341.744213:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961505408:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078341.744235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078341.744248:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078341/real 1769078341] req@ffff895e44e28a80 x1855008961505408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078341.744262:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961505408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078341.744284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078341.744287:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078341.744300:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961505408:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078341.744306:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961505536:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078341.744326:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078341.744337:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078341/real 1769078341] req@ffff895e44e29500 x1855008961505536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078341.744349:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961505536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078341.744356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078341.744360:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078341.744363:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961505536:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078342.255846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078342.447957:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078343.279919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078343.471981:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078344.303909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078344.495912:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078345.327886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078345.519923:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078346.351853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078346.543935:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078346.863914:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078346.863923:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078346.863926:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078346.863935:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4029 00000100:00080000:3.0:1769078346.863944:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078346.863949:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078346.863978:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961505664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078346.864015:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078346.864019:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078346.864020:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078346.864024:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4029 00000100:00080000:3.0:1769078346.864028:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078346.864032:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078346.864041:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078346.864052:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961505792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078346.864060:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078346.864061:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961505664:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078346.864063:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078346.864065:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078346.864069:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4029 00000100:00080000:3.0:1769078346.864072:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078346.864075:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078346.864079:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078346.864086:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961505920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078346.864121:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078346.864157:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078346/real 1769078346] req@ffff895e453d4380 x1855008961505664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078346.864170:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961505664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078346.864177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078346.864180:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078346.864182:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961505664:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078346.864199:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961505792:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078346.864213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078346.864220:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078346/real 1769078346] req@ffff895e453d5180 x1855008961505792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078346.864227:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961505792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078346.864234:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078346.864237:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078346.864240:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961505792:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078346.864245:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961505920:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078346.864264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078346.864273:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078346/real 1769078346] req@ffff895e453d5880 x1855008961505920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078346.864295:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961505920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078346.864303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078346.864305:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078346.864309:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961505920:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078347.375870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078347.567909:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078348.399957:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078348.591979:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078349.423908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078349.615915:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078350.447866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078350.639940:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078351.471893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078351.663909:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078351.983876:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078351.983886:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078351.983890:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078351.983899:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4034 00000100:00080000:0.0:1769078351.983909:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078351.983914:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078351.983943:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961506048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078351.983970:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078351.983973:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078351.983974:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078351.983977:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4034 00000100:00080000:0.0:1769078351.983980:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078351.983984:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078351.983993:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078351.984007:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961506048:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078351.984021:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961506176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078351.984033:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078351.984036:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078351.984043:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078351.984048:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4034 00000100:00080000:0.0:1769078351.984053:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078351.984057:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078351.984062:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769078351.984070:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078351.984076:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961506304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078351.984108:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078351/real 1769078351] req@ffff895e4383c000 x1855008961506048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078351.984128:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961506048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078351.984137:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078351.984140:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078351.984143:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961506048:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078351.984160:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961506176:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078351.984176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078351.984184:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078351/real 1769078351] req@ffff895e4383f480 x1855008961506176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078351.984192:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961506176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078351.984197:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078351.984199:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078351.984201:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961506176:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078351.984204:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961506304:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078351.984213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078351.984218:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078351/real 1769078351] req@ffff895e4383ce00 x1855008961506304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078351.984235:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961506304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078351.984240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078351.984242:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078351.984243:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961506304:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078352.495942:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078352.687896:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078353.519854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078353.711908:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078354.543839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078354.735889:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078355.567845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078355.759890:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078356.591868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078356.783935:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078357.103847:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078357.103862:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078357.103867:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078357.103879:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4039 00000100:00080000:0.0:1769078357.103894:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078357.103903:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078357.103952:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961506432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078357.103991:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078357.103995:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078357.103997:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078357.104001:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4039 00000100:00080000:0.0:1769078357.104027:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078357.104033:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078357.104044:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078357.104053:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961506432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078357.104061:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961506560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078357.104073:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078357.104077:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078357.104079:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078357.104083:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4039 00000100:00080000:0.0:1769078357.104087:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078357.104091:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078357.104096:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078357.104117:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961506688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078357.104139:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078357.104188:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078357/real 1769078357] req@ffff895e4383c380 x1855008961506432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078357.104209:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961506432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078357.104220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078357.104224:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078357.104229:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961506432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078357.104256:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961506560:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078357.104277:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078357.104290:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078357/real 1769078357] req@ffff895e4383f100 x1855008961506560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078357.104302:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961506560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078357.104323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078357.104328:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078357.104331:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961506560:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078357.104337:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961506688:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078357.104358:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078357.104367:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078357/real 1769078357] req@ffff895e4383fb80 x1855008961506688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078357.104390:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961506688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078357.104398:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078357.104401:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078357.104404:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961506688:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078357.615895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078357.807975:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078358.639930:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078358.832592:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078359.664779:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078359.855961:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078360.687847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078360.879947:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078361.711945:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078361.903865:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078362.223858:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078362.223867:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078362.223871:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078362.223880:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4044 00000100:00080000:0.0:1769078362.223891:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078362.223898:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078362.223956:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961506816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078362.223985:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078362.223988:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078362.223990:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078362.223993:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4044 00000100:00080000:0.0:1769078362.223997:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078362.224001:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078362.224009:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078362.224024:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961506944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078362.224028:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961506816:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078362.224037:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078362.224051:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078362.224052:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078362.224059:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4044 00000100:00080000:0.0:1769078362.224063:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078362.224067:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078362.224072:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078362.224084:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961507072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078362.224097:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078362.224148:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078362/real 1769078362] req@ffff895e4383c700 x1855008961506816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078362.224166:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961506816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078362.224176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078362.224180:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078362.224190:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961506816:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078362.224218:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961506944:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078362.224240:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078362.224252:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078362/real 1769078362] req@ffff895e4383e680 x1855008961506944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078362.224265:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961506944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078362.224273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078362.224276:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078362.224279:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961506944:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078362.224294:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961507072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078362.224315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078362.224325:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078362/real 1769078362] req@ffff895e4383d880 x1855008961507072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078362.224336:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961507072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078362.224343:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078362.224346:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078362.224349:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961507072:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078362.735886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078362.927897:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078363.759860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078363.951985:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078364.783905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078364.975936:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078365.807853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078365.999907:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078366.831833:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078367.023888:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078367.343841:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078367.343848:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078367.343864:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078367.343873:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4049 00000100:00080000:0.0:1769078367.343883:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078367.343890:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078367.343922:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961507200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078367.343951:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078367.343953:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078367.343955:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078367.343957:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4049 00000100:00080000:0.0:1769078367.343960:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078367.343963:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078367.343969:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078367.343979:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961507328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078367.343987:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078367.343989:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078367.343990:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769078367.343995:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961507200:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078367.343996:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4049 00000100:00080000:0.0:1769078367.344000:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078367.344005:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078367.344008:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078367.344015:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961507456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769078367.344051:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078367.344094:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078367/real 1769078367] req@ffff895e4383d180 x1855008961507200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078367.344111:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961507200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078367.344118:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078367.344121:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078367.344124:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961507200:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078367.344140:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961507328:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078367.344156:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078367.344163:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078367/real 1769078367] req@ffff895e4383df80 x1855008961507328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078367.344170:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961507328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078367.344175:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078367.344177:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078367.344179:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961507328:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078367.344182:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961507456:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078367.344190:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078367.344195:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078367/real 1769078367] req@ffff895e4383ed80 x1855008961507456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078367.344201:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961507456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078367.344214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078367.344216:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078367.344218:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961507456:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078367.855810:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078368.047919:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078368.879885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078369.071946:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078369.904411:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078370.095917:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078370.927901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078371.119963:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078371.951841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078372.143979:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078372.463868:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078372.463876:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078372.463879:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078372.463888:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4055 00000100:00080000:3.0:1769078372.463899:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078372.463906:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078372.463940:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961507584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078372.463971:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078372.463973:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078372.463974:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078372.463977:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4055 00000100:00080000:3.0:1769078372.463980:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078372.463983:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078372.463989:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078372.463998:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961507712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078372.464005:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078372.464008:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769078372.464014:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961507584:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078372.464023:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078372.464028:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4055 00000100:00080000:3.0:1769078372.464032:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078372.464036:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078372.464042:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078372.464054:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961507840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078372.464074:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078372.464107:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078372/real 1769078372] req@ffff895e453d4700 x1855008961507584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078372.464125:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961507584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078372.464135:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078372.464138:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078372.464142:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961507584:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078372.464164:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961507712:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078372.464182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078372.464192:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078372/real 1769078372] req@ffff895e453d7100 x1855008961507712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078372.464203:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961507712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078372.464211:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078372.464214:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078372.464217:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961507712:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078372.464222:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961507840:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078372.464237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078372.464253:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078372/real 1769078372] req@ffff895e453d7b80 x1855008961507840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078372.464264:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961507840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078372.464273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078372.464276:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078372.464279:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961507840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078372.975858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078373.168136:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078373.999896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078374.191877:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078375.024450:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078375.215922:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078376.048030:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078376.239968:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078377.071858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078377.263880:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078377.583923:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078377.583932:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078377.583935:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078377.583942:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4060 00000100:00080000:2.0:1769078377.583952:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078377.583958:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078377.583991:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961507968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078377.584032:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078377.584037:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078377.584052:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078377.584057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4060 00000100:00080000:2.0:1769078377.584062:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078377.584066:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078377.584073:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961507968:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769078377.584075:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078377.584092:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961508096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078377.584104:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078377.584107:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078377.584109:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078377.584113:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4060 00000100:00080000:2.0:1769078377.584117:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078377.584120:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078377.584125:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078377.584136:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961508224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078377.584141:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078377.584182:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078377/real 1769078377] req@ffff895e44e29f80 x1855008961507968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078377.584198:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961507968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078377.584207:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078377.584212:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078377.584215:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961507968:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078377.584237:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961508096:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078377.584256:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078377.584267:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078377/real 1769078377] req@ffff895e44e28e00 x1855008961508096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078377.584277:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961508096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078377.584309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078377.584314:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078377.584317:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961508096:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078377.584323:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961508224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078377.584343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078377.584353:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078377/real 1769078377] req@ffff895e44e2b480 x1855008961508224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078377.584363:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961508224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078377.584370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078377.584373:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078377.584376:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961508224:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078378.095861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078378.287903:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078379.119862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078379.311855:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078380.143911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078380.335979:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078381.167877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078381.359906:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078382.191892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078382.383982:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078382.703881:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078382.703891:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078382.703895:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078382.703904:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4065 00000100:00080000:3.0:1769078382.703915:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078382.703922:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078382.703955:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961508352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078382.703986:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078382.703990:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078382.703992:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078382.703996:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4065 00000100:00080000:3.0:1769078382.704000:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078382.704004:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078382.704012:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078382.704024:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961508352:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078382.704027:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961508480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078382.704060:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078382.704064:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078382.704066:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078382.704070:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4065 00000100:00080000:3.0:1769078382.704075:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078382.704078:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:0.0:1769078382.704084:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:3.0:1769078382.704084:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078382.704107:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961508608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078382.704144:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078382/real 1769078382] req@ffff895e453d5500 x1855008961508352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078382.704164:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961508352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078382.704185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078382.704189:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078382.704200:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961508352:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078382.704235:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961508480:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078382.704256:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078382.704267:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078382/real 1769078382] req@ffff895e453d4e00 x1855008961508480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078382.704290:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961508480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078382.704298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078382.704301:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078382.704304:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961508480:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078382.704309:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961508608:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078382.704328:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078382.704352:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078382/real 1769078382] req@ffff895e453d4a80 x1855008961508608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078382.704364:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961508608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078382.704371:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078382.704374:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078382.704377:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961508608:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078383.215854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078383.407917:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078384.239858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078384.431891:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078385.263836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078385.455892:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078386.287848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078386.479913:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078387.311846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078387.503902:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078387.823828:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078387.823837:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078387.823840:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078387.823847:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4070 00000100:00080000:3.0:1769078387.823856:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078387.823861:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078387.823897:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961508736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078387.823919:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078387.823921:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078387.823922:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078387.823925:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4070 00000100:00080000:3.0:1769078387.823927:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078387.823929:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078387.823951:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078387.823961:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961508864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078387.823967:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961508736:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078387.823981:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078387.823984:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078387.823985:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078387.823988:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4070 00000100:00080000:3.0:1769078387.823991:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078387.823994:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078387.823997:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078387.824005:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961508992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078387.824022:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078387.824066:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078387/real 1769078387] req@ffff895e453d7480 x1855008961508736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078387.824078:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961508736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078387.824084:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078387.824087:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078387.824090:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961508736:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078387.824106:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961508864:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078387.824119:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078387.824140:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078387/real 1769078387] req@ffff895e453d5f80 x1855008961508864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078387.824147:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961508864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078387.824153:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078387.824161:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078387.824163:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961508864:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078387.824173:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961508992:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078387.824199:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078387.824206:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078387/real 1769078387] req@ffff895e453d5c00 x1855008961508992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078387.824213:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961508992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078387.824217:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078387.824219:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078387.824221:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961508992:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078388.335857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078388.527886:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078389.359837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078389.551910:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078390.383894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078390.575963:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078391.407850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078391.600025:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078392.431862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078392.623904:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078392.944662:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078392.944676:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078392.944681:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078392.944693:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4075 00000100:00080000:3.0:1769078392.944707:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078392.944715:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078392.944777:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961509120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078392.944814:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078392.944818:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078392.944819:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078392.944823:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4075 00000100:00080000:3.0:1769078392.944844:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078392.944849:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078392.944860:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078392.944874:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961509248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078392.944884:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078392.944887:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078392.944889:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078392.944893:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4075 00000100:00080000:3.0:1769078392.944895:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078392.944898:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078392.944902:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078392.944910:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961509376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078392.945024:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961509120:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078392.945121:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078392.945179:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078392/real 1769078392] req@ffff895e453d7800 x1855008961509120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078392.945198:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961509120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078392.945208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078392.945212:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078392.945216:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961509120:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078392.945230:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961509248:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078392.945253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078392.945265:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078392/real 1769078392] req@ffff895e453d5c00 x1855008961509248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078392.945276:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961509248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078392.945284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078392.945287:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078392.945291:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961509248:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078392.945296:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961509376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078392.945312:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078392.945321:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078392/real 1769078392] req@ffff895e453d5f80 x1855008961509376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078392.945331:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961509376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078392.945339:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078392.945343:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078392.945346:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961509376:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078393.455908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078393.648580:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078394.479919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078394.672123:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078395.504201:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078395.696248:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078396.527934:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078396.721273:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078397.551861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078397.744026:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078398.063874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078398.063885:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078398.063890:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078398.063900:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4080 00000100:00080000:2.0:1769078398.063911:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078398.063919:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078398.063964:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961509504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078398.063991:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078398.063994:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078398.063996:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078398.064000:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4080 00000100:00080000:2.0:1769078398.064003:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078398.064019:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078398.064032:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078398.064044:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961509504:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078398.064048:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961509632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078398.064061:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078398.064066:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078398.064068:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078398.064071:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4080 00000100:00080000:2.0:1769078398.064076:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078398.064079:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078398.064084:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078398.064095:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961509760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078398.064263:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961509632:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078398.064297:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961509760:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078398.575868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078398.768180:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078399.599890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078399.792358:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078400.623858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078400.816246:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769078401.136046:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769078401.136080:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769078401.136092:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769078401.136128:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769078401.136137:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769078401.136217:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078398/real 1769078401] req@ffff895e44e2b100 x1855008961509632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078401.136240:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961509632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078401.136251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078401.136256:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078401.136261:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961509632:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078401.136269:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078398/real 1769078401] req@ffff895e44e2a300 x1855008961509760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078401.136280:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961509760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078401.136285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078401.136288:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078401.136291:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961509760:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078401.136297:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078398/real 1769078401] req@ffff895e44e2aa00 x1855008961509504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078401.136308:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961509504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078401.136329:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078401.136336:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078401.136340:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961509504:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078401.648198:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078401.839956:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078402.671921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078402.864423:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078403.184013:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078403.184029:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078403.184034:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078403.184047:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4085 00000100:00080000:3.0:1769078403.184061:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078403.184069:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078403.184112:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961509888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078403.184175:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078403.184256:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078403.184258:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078403.184263:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4085 00000100:00080000:3.0:1769078403.184267:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078403.184272:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078403.184283:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078403.184296:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961510016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078403.184306:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078403.184309:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078403.184311:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078403.184314:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4085 00000100:00080000:3.0:1769078403.184317:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078403.184319:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078403.184323:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078403.184331:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961510144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078403.184436:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961509888:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078403.184514:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078403.184605:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078403/real 1769078403] req@ffff895e453d7480 x1855008961509888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078403.184626:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961509888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078403.184636:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078403.184640:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078403.184645:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961509888:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078403.184652:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961510016:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078403.184677:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078403.184690:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078403/real 1769078403] req@ffff895e453d4a80 x1855008961510016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078403.184700:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961510016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078403.184707:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078403.184719:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078403.184722:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961510016:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078403.184728:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961510144:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078403.184784:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078403.184795:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078403/real 1769078403] req@ffff895e453d4e00 x1855008961510144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078403.184805:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961510144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078403.184812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078403.184815:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078403.184817:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961510144:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078403.696247:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078403.888054:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078404.720226:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078404.912128:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078405.744112:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078405.936016:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078406.767948:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078406.959990:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078407.792154:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078407.984014:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078408.303884:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078408.303895:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078408.303899:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078408.303908:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4090 00000100:00080000:3.0:1769078408.303919:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078408.303926:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078408.303958:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961510272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078408.304001:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078408.304005:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078408.304007:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078408.304016:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4090 00000100:00080000:3.0:1769078408.304020:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078408.304024:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078408.304033:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078408.304048:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961510400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078408.304060:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078408.304061:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961510272:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078408.304064:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078408.304068:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078408.304074:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4090 00000100:00080000:3.0:1769078408.304078:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078408.304082:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078408.304087:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078408.304099:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961510528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078408.304124:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078408.304157:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078408/real 1769078408] req@ffff895e453d5500 x1855008961510272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078408.304174:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961510272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078408.304184:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078408.304188:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078408.304198:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961510272:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078408.304220:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961510400:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078408.304242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078408.304267:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078408/real 1769078408] req@ffff895e453d7b80 x1855008961510400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078408.304280:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961510400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078408.304288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078408.304291:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078408.304294:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961510400:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078408.304300:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961510528:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078408.304317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078408.304327:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078408/real 1769078408] req@ffff895e453d7100 x1855008961510528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078408.304337:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961510528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078408.304345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078408.304348:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078408.304351:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961510528:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078408.820072:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078409.007940:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078409.839866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078410.032003:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078410.863859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078411.055968:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078411.887874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078412.080768:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078412.911841:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078413.103903:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078413.423949:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078413.423961:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078413.423965:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078413.423975:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4095 00000100:00080000:2.0:1769078413.423986:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078413.423994:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078413.424029:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961510656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078413.424103:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961510656:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078413.424179:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078413.424183:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078413.424185:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078413.424190:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4095 00000800:00000100:0.0:1769078413.424195:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:2.0:1769078413.424196:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078413.424203:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078413.424213:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078413.424229:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961510784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078413.424236:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078413/real 1769078413] req@ffff895e44e28000 x1855008961510656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078413.424243:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078413.424252:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078413.424256:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078413.424261:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4095 00000100:00080000:2.0:1769078413.424265:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769078413.424267:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961510656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078413.424279:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00080000:0.0:1769078413.424284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1769078413.424286:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1769078413.424291:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078413.424297:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961510656:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078413.424301:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961510912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078413.424325:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961510784:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078413.424348:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078413.424360:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078413/real 1769078413] req@ffff895e44e28380 x1855008961510784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078413.424539:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961510784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078413.424550:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078413.424553:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078413.424557:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961510784:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078413.424564:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961510912:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078413.424595:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078413.424606:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078413/real 1769078413] req@ffff895e44e29880 x1855008961510912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078413.424632:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961510912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078413.424640:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078413.424643:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078413.424646:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961510912:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078413.935858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078414.128303:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078414.959886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078415.152737:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078415.986434:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078416.175956:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078417.007945:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078417.199948:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078418.031913:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078418.223987:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078418.543907:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078418.543919:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078418.543924:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078418.543935:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4101 00000100:00080000:3.0:1769078418.543949:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078418.543958:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078418.544009:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961511040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078418.544046:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078418.544050:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078418.544051:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078418.544056:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4101 00000100:00080000:3.0:1769078418.544059:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078418.544064:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078418.544074:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078418.544087:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961511168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078418.544095:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078418.544097:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078418.544099:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078418.544101:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4101 00000100:00080000:3.0:1769078418.544104:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078418.544108:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078418.544109:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961511040:192.168.204.117@tcp:250:kworker.0 00000008:00100000:3.0:1769078418.544134:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078418.544147:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961511296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078418.544202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078418.544242:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078418/real 1769078418] req@ffff895e453d4700 x1855008961511040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078418.544256:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961511040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078418.544264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078418.544267:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078418.544270:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961511040:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078418.544275:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961511168:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078418.544290:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078418.544297:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078418/real 1769078418] req@ffff895e453d5880 x1855008961511168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078418.544304:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961511168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078418.544315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078418.544317:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078418.544319:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961511168:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078418.544342:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961511296:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078418.544354:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078418.544360:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078418/real 1769078418] req@ffff895e453d5180 x1855008961511296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078418.544370:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961511296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078418.544375:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078418.544377:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078418.544379:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961511296:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078419.055862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078419.247935:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078420.079835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078420.271940:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078421.103902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078421.296720:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078422.127914:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078422.320054:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078423.151901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078423.343910:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078423.663839:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078423.663848:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078423.663853:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078423.663861:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4106 00000100:00080000:3.0:1769078423.663871:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078423.663877:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078423.663905:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961511424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078423.663926:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078423.663929:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078423.663930:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078423.663932:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4106 00000100:00080000:3.0:1769078423.663934:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078423.663936:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078423.663942:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078423.663949:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961511552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078423.663956:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078423.663958:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078423.663959:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078423.663961:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4106 00000100:00100000:0.0:1769078423.663972:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961511424:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078423.663980:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078423.663985:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078423.663988:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078423.663999:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961511680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078423.664031:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078423.664071:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078423/real 1769078423] req@ffff895e453d4380 x1855008961511424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078423.664087:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961511424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078423.664107:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078423.664112:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078423.664116:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961511424:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078423.664122:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961511552:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078423.664142:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078423.664152:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078423/real 1769078423] req@ffff895e453d6680 x1855008961511552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078423.664163:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961511552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078423.664171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078423.664174:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078423.664176:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961511552:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078423.664196:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961511680:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078423.664210:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078423.664218:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078423/real 1769078423] req@ffff895e453d6300 x1855008961511680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078423.664229:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961511680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078423.664236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078423.664239:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078423.664242:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961511680:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078424.176443:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078424.367952:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078425.200070:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078425.391972:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078426.223860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078426.415909:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078427.247845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078427.439845:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078428.272035:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078428.464138:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078428.783942:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078428.783956:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078428.783961:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078428.783974:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4111 00000100:00080000:2.0:1769078428.784012:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078428.784021:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078428.784065:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961511808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078428.784101:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078428.784105:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078428.784106:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078428.784110:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4111 00000100:00080000:2.0:1769078428.784114:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078428.784118:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078428.784129:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078428.784143:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961511936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078428.784155:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961511808:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078428.784157:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078428.784164:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078428.784168:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078428.784172:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4111 00000100:00080000:2.0:1769078428.784176:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078428.784179:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078428.784185:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078428.784196:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961512064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078428.784241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078428.784280:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078428/real 1769078428] req@ffff895e44e29880 x1855008961511808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078428.784295:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961511808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078428.784303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078428.784306:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078428.784309:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961511808:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078428.784333:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961511936:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078428.784347:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078428.784354:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078428/real 1769078428] req@ffff895e44e28380 x1855008961511936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078428.784361:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961511936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078428.784367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078428.784369:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078428.784371:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961511936:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078428.784375:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961512064:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078428.784390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078428.784408:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078428/real 1769078428] req@ffff895e44e28000 x1855008961512064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078428.784421:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961512064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078428.784426:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078428.784429:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078428.784431:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961512064:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078429.295987:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078429.487881:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078430.321181:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078430.511988:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078431.343832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078431.535858:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078432.367835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078432.559921:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078433.392036:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078433.583899:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078433.903907:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078433.903923:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078433.903928:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078433.903937:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4116 00000100:00080000:2.0:1769078433.903952:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078433.903958:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078433.904007:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961512192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078433.904042:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078433.904045:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078433.904047:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078433.904051:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4116 00000100:00080000:2.0:1769078433.904054:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078433.904066:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078433.904079:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078433.904099:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961512320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078433.904110:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078433.904121:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769078433.904124:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961512192:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078433.904125:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078433.904147:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4116 00000100:00080000:2.0:1769078433.904152:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078433.904179:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078433.904187:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078433.904211:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961512448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078433.904239:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078433.904319:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078433/real 1769078433] req@ffff895e44e2aa00 x1855008961512192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078433.904341:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961512192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078433.904368:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078433.904374:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078433.904378:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961512192:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078433.904412:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961512320:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078433.904459:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078433.904482:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078433/real 1769078433] req@ffff895e44e2a300 x1855008961512320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078433.904494:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961512320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078433.904502:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078433.904506:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078433.904509:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961512320:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078433.904515:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961512448:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078433.904534:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078433.904544:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078433/real 1769078433] req@ffff895e44e2b100 x1855008961512448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078433.904555:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961512448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078433.904562:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078433.904566:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078433.904569:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961512448:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078434.415938:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078434.607981:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078435.439961:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078435.631950:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078436.463896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078436.655943:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078437.488385:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078437.679947:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078438.511934:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078438.703946:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078439.024106:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078439.024136:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078439.024142:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078439.024154:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4121 00000100:00080000:1.0:1769078439.024167:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078439.024174:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078439.024222:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961512576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078439.024354:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961512576:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078439.024429:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:1.0:1769078439.024507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078439.024513:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078439.024515:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078439.024540:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4121 00000100:00080000:1.0:1769078439.024546:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078439.024551:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078439.024562:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078439.024577:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961512704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078439.024620:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078439.024625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078439.024627:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078439.024632:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4121 00000100:00080000:1.0:1769078439.024637:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078439.024642:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078439.024648:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078439.024655:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078439/real 1769078439] req@ffff895e4713d180 x1855008961512576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078439.024668:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961512832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078439.024681:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961512576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078439.024694:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078439.024699:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078439.024702:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961512576:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078439.024735:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961512704:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078439.024809:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078439.024822:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078439/real 1769078439] req@ffff895e4713c380 x1855008961512704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078439.024835:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961512704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078439.024843:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078439.024847:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078439.024850:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961512704:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078439.024857:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961512832:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078439.024877:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078439.024888:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078439/real 1769078439] req@ffff895e4713df80 x1855008961512832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078439.024914:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961512832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078439.024921:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078439.024924:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078439.024927:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961512832:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078439.535940:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078439.729522:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078440.560772:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078440.752781:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078441.583945:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078441.780282:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078442.610236:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078442.799924:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1769078443.631874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078443.823927:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078444.143856:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078444.143870:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078444.143875:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078444.143887:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4126 00000100:00080000:3.0:1769078444.143901:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078444.143908:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078444.143952:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961512960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078444.143986:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078444.143989:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078444.143991:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078444.143995:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4126 00000100:00080000:3.0:1769078444.143999:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078444.144002:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078444.144013:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078444.144027:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961513088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078444.144039:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769078444.144048:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961512960:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078444.144059:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078444.144061:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078444.144066:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4126 00000100:00080000:3.0:1769078444.144070:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078444.144073:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078444.144078:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078444.144091:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961513216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078444.144143:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078444.144200:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078444/real 1769078444] req@ffff895e453d6d80 x1855008961512960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078444.144220:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961512960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078444.144230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078444.144234:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078444.144238:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961512960:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078444.144265:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961513088:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078444.144285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078444.144313:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078444/real 1769078444] req@ffff895e453d6300 x1855008961513088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078444.144326:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961513088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078444.144334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078444.144337:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078444.144341:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961513088:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078444.144347:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961513216:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078444.144363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078444.144371:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078444/real 1769078444] req@ffff895e453d6680 x1855008961513216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078444.144382:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961513216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078444.144389:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078444.144393:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078444.144396:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961513216:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078444.656680:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078444.848004:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078445.679901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078445.873891:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078446.704253:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078446.896304:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078447.728237:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078447.920197:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078448.752293:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078448.943933:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078449.268050:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078449.268060:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078449.268064:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078449.268082:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4131 00000100:00080000:0.0:1769078449.268101:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078449.268124:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078449.268155:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961513344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078449.268184:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078449.268187:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078449.268189:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078449.268193:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4131 00000100:00080000:0.0:1769078449.268196:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078449.268200:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078449.268208:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078449.268217:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961513472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078449.268226:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078449.268228:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078449.268230:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078449.268241:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4131 00000100:00080000:0.0:1769078449.268244:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078449.268248:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078449.268252:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078449.268259:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961513600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078449.268328:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961513344:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078449.268380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078449.268411:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078449/real 1769078449] req@ffff895e4383ed80 x1855008961513344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078449.268427:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961513344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078449.268436:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078449.268439:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078449.268443:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961513344:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078449.268449:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961513472:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078449.268464:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078449.268472:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078449/real 1769078449] req@ffff895e4383df80 x1855008961513472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078449.268483:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961513472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078449.268496:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078449.268499:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078449.268502:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961513472:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078449.268507:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961513600:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078449.268521:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078449.268529:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078449/real 1769078449] req@ffff895e4383d180 x1855008961513600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078449.268539:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961513600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078449.268546:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078449.268549:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078449.268551:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961513600:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078449.775935:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078449.968020:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078450.799904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078450.991950:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078451.823863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078452.015967:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078452.847839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078453.040090:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078453.872387:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078454.066227:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078454.385262:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078454.385272:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078454.385277:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078454.385286:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4136 00000100:00080000:1.0:1769078454.385296:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078454.385303:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078454.385339:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961513728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078454.385367:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078454.385383:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078454.385386:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078454.385390:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4136 00000100:00080000:1.0:1769078454.385395:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078454.385400:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078454.385409:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078454.385415:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961513728:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078454.385426:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961513856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078454.385439:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078454.385442:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078454.385445:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078454.385449:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4136 00000100:00080000:1.0:1769078454.385453:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078454.385456:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078454.385461:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769078454.385473:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078454.385475:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961513984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078454.385511:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078454/real 1769078454] req@ffff895e4713dc00 x1855008961513728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078454.385528:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961513728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078454.385539:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078454.385542:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078454.385563:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961513728:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078454.385585:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961513856:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078454.385605:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078454.385615:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078454/real 1769078454] req@ffff895e4713f800 x1855008961513856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078454.385627:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961513856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078454.385634:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078454.385637:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078454.385640:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961513856:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078454.385646:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961513984:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078454.385661:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078454.385671:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078454/real 1769078454] req@ffff895e4713ca80 x1855008961513984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078454.385681:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961513984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078454.385689:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078454.385692:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078454.385695:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961513984:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078454.895865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078455.089095:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078455.919882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078456.111918:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078456.943855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078457.136217:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078457.967852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078458.159989:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078458.991922:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078459.185500:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078459.503891:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078459.503900:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078459.503904:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078459.503914:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4142 00000100:00080000:0.0:1769078459.503937:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078459.503944:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078459.503980:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961514112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078459.504012:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078459.504016:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078459.504018:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078459.504022:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4142 00000100:00080000:0.0:1769078459.504026:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078459.504030:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078459.504038:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078459.504049:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961514240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078459.504058:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078459.504061:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078459.504062:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078459.504066:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4142 00000100:00080000:0.0:1769078459.504069:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078459.504072:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078459.504091:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078459.504101:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961514368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078459.505602:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961514112:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078459.505653:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078459.505696:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078459/real 1769078459] req@ffff895e4383d880 x1855008961514112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078459.505715:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961514112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078459.505724:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078459.505728:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078459.505733:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961514112:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078459.505739:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961514240:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078459.505795:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078459.505808:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078459/real 1769078459] req@ffff895e4383e680 x1855008961514240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078459.505819:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961514240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078459.505827:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078459.505830:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078459.505833:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961514240:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078459.505838:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961514368:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078459.505856:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078459.505866:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078459/real 1769078459] req@ffff895e4383c700 x1855008961514368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078459.505877:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961514368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078459.505898:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078459.505902:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078459.505912:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961514368:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078460.015851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078460.208595:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078461.039986:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078461.231928:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078462.063848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078462.256217:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078463.087878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078463.279929:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078464.111851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078464.303934:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078464.623986:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078464.623996:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078464.624000:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078464.624009:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4147 00000100:00080000:0.0:1769078464.624019:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078464.624026:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078464.624060:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961514496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078464.624082:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078464.624090:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078464.624091:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078464.624097:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4147 00000100:00080000:0.0:1769078464.624100:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078464.624102:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078464.624107:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961514496:192.168.204.117@tcp:250:kworker.0 00000008:00100000:0.0:1769078464.624112:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078464.624122:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961514624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078464.624129:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078464.624143:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078464.624144:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078464.624147:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4147 00000100:00080000:0.0:1769078464.624151:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078464.624153:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078464.624157:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078464.624165:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961514752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078464.624201:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961514624:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078464.624220:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961514752:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078465.135879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078465.328116:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078466.159932:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078466.351955:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078467.183894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078467.376000:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769078467.700414:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769078467.700473:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078467.700495:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078467.700551:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078467.700562:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:1.0:1769078467.700641:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078464/real 1769078467] req@ffff895e4383f100 x1855008961514624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078467.700676:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961514624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078467.700688:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078467.700693:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078467.700698:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961514624:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078467.700705:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078464/real 1769078467] req@ffff895e4383c380 x1855008961514752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078467.700717:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961514752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078467.700725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078467.700729:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078467.700732:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961514752:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078467.700739:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078464/real 1769078467] req@ffff895e4383fb80 x1855008961514496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078467.700802:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961514496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078467.700809:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078467.700813:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078467.700816:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961514496:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078468.207925:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078468.401010:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078469.232070:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078469.424504:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078469.746374:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078469.746387:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078469.746391:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078469.746401:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4152 00000100:00080000:3.0:1769078469.746420:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078469.746431:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078469.746470:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961514880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078469.746733:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078469.746737:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078469.746739:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078469.746802:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961514880:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078469.746880:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4152 00000800:00000100:1.0:1769078469.746884:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769078469.746887:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078469.746894:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078469.746905:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078469.746920:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961515008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078469.746929:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078469/real 1769078469] req@ffff895e453d4380 x1855008961514880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078469.746932:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078469.746940:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078469.746951:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1769078469.746953:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961514880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078469.746958:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4152 00000100:00080000:3.0:1769078469.746966:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078469.746970:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078469.746973:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078469.746980:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1769078469.746990:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078469.746995:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961515136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078469.746997:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961514880:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078469.747028:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961515008:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078469.747052:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078469.747062:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078469/real 1769078469] req@ffff895e453d5180 x1855008961515008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078469.747073:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961515008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078469.747081:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078469.747083:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078469.747086:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961515008:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078469.747091:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961515136:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078469.747106:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078469.747116:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078469/real 1769078469] req@ffff895e453d5880 x1855008961515136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078469.747139:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961515136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078469.747147:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078469.747149:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078469.747152:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961515136:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078470.255854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078470.448006:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078471.279853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078471.471958:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078472.303886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078472.495966:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078473.327876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078473.519888:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078474.351887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078474.543940:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078474.863825:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078474.863837:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078474.863842:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078474.863853:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4157 00000100:00080000:3.0:1769078474.863866:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078474.863879:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078474.863921:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961515264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078474.863947:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078474.863954:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078474.863955:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078474.863959:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4157 00000100:00080000:3.0:1769078474.863962:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078474.863966:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078474.863977:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078474.863987:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961515392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078474.863999:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961515264:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078474.864028:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078474.864031:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078474.864033:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078474.864038:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4157 00000100:00080000:3.0:1769078474.864043:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078474.864047:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078474.864052:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078474.864061:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961515520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078474.864079:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078474.864136:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078474/real 1769078474] req@ffff895e453d4700 x1855008961515264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078474.864154:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961515264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078474.864164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078474.864167:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078474.864177:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961515264:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078474.864199:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961515392:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078474.864218:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078474.864228:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078474/real 1769078474] req@ffff895e453d7100 x1855008961515392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078474.864236:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961515392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078474.864254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078474.864257:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078474.864259:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961515392:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078474.864264:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961515520:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078474.864279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078474.864286:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078474/real 1769078474] req@ffff895e453d7b80 x1855008961515520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078474.864293:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961515520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078474.864299:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078474.864301:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078474.864303:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961515520:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078475.376473:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078475.567935:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078476.399835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078476.591942:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078477.423828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078477.615883:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078478.447857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078478.639947:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078479.471852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078479.663881:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078479.983852:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078479.983859:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078479.983863:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078479.983871:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4162 00000100:00080000:3.0:1769078479.983879:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078479.983885:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078479.983916:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961515648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078479.983943:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078479.983947:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078479.983949:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078479.983952:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4162 00000100:00080000:3.0:1769078479.983956:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078479.983959:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078479.983981:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078479.983991:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961515648:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078479.983994:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961515776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078479.984005:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078479.984007:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078479.984008:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078479.984012:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4162 00000100:00080000:3.0:1769078479.984015:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078479.984017:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078479.984020:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078479.984029:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961515904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078479.984060:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078479.984111:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078479/real 1769078479] req@ffff895e453d5500 x1855008961515648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078479.984131:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961515648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078479.984147:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078479.984150:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078479.984160:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961515648:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078479.984182:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961515776:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078479.984204:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078479.984217:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078479/real 1769078479] req@ffff895e453d4e00 x1855008961515776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078479.984227:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961515776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078479.984232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078479.984234:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078479.984236:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961515776:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078479.984240:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961515904:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078479.984252:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078479.984258:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078479/real 1769078479] req@ffff895e453d4a80 x1855008961515904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078479.984269:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961515904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078479.984283:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078479.984286:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078479.984289:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961515904:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078480.501867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078480.688294:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078481.520084:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078481.715150:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078482.545441:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078482.736419:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078483.567863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078483.759950:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078484.591866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078484.784453:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078485.105418:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078485.105429:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078485.105433:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078485.105440:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4167 00000100:00080000:3.0:1769078485.105450:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078485.105457:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078485.105505:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961516032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078485.105538:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078485.105541:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078485.105542:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078485.105545:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4167 00000100:00080000:3.0:1769078485.105563:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078485.105568:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078485.105576:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078485.105590:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961516160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078485.105599:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078485.105602:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078485.105604:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078485.105608:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4167 00000100:00080000:3.0:1769078485.105611:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078485.105614:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078485.105618:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078485.105629:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961516288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078485.105796:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961516032:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078485.105859:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078485.105897:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078485/real 1769078485] req@ffff895e453d7480 x1855008961516032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078485.105984:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961516032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078485.105994:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078485.105998:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078485.106002:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961516032:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078485.106009:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961516160:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078485.106032:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078485.106045:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078485/real 1769078485] req@ffff895e453d5f80 x1855008961516160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078485.106057:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961516160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078485.106065:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078485.106068:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078485.106071:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961516160:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078485.106077:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961516288:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078485.106095:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078485.106106:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078485/real 1769078485] req@ffff895e453d5c00 x1855008961516288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078485.106126:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961516288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078485.106133:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078485.106137:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078485.106152:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961516288:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078485.617271:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078485.807962:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078486.639858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078486.831929:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078487.663872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078487.855911:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078488.688063:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078488.879907:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078489.714070:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078489.904078:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078490.223938:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078490.223948:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078490.223952:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078490.223961:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4172 00000100:00080000:2.0:1769078490.223971:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078490.223978:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078490.224014:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961516416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078490.224082:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961516416:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078490.224141:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:2.0:1769078490.224171:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078490.224175:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078490.224177:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078490.224181:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4172 00000100:00080000:2.0:1769078490.224185:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078490.224189:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078490.224198:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078490.224211:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961516544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078490.224289:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078490/real 1769078490] req@ffff895e44e2b480 x1855008961516416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078490.224307:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961516416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078490.224316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078490.224320:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078490.224324:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961516416:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078490.224348:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961516544:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078490.224371:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078490.224383:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078490/real 1769078490] req@ffff895e44e28e00 x1855008961516544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078490.224395:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961516544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078490.224427:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078490.224430:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078490.224434:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961516544:192.168.204.117@tcp:38:kworker.0 00000100:00080000:2.0:1769078490.224630:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078490.224634:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078490.224637:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078490.224932:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4172 00000100:00080000:2.0:1769078490.224937:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078490.224941:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078490.224947:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078490.224959:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961516672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078490.225007:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961516672:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078490.225031:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078490.225090:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078490/real 1769078490] req@ffff895e44e29f80 x1855008961516672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078490.225103:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961516672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078490.225111:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078490.225114:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078490.225118:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961516672:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078490.735853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078490.927925:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078491.760537:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078491.951913:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078492.783871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078492.976019:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078493.807862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078494.000012:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078494.831868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078495.025452:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078495.343983:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078495.344005:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078495.344008:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078495.344016:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4177 00000100:00080000:0.0:1769078495.344025:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078495.344030:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078495.344060:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961516800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078495.344083:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078495.344085:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078495.344086:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078495.344089:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4177 00000100:00080000:0.0:1769078495.344091:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078495.344094:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078495.344101:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078495.344109:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961516928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078495.344115:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078495.344117:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078495.344118:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078495.344120:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4177 00000100:00080000:0.0:1769078495.344121:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078495.344123:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078495.344125:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078495.344130:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961517056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078495.344251:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961516800:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078495.344309:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078495.344356:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078495/real 1769078495] req@ffff895e4383ce00 x1855008961516800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078495.344374:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961516800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078495.344384:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078495.344388:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078495.344398:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961516800:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078495.344406:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961516928:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078495.344438:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078495.344451:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078495/real 1769078495] req@ffff895e4383f480 x1855008961516928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078495.344462:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961516928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078495.344469:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078495.344471:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078495.344474:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961516928:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078495.344479:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961517056:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078495.344494:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078495.344503:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078495/real 1769078495] req@ffff895e4383c000 x1855008961517056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078495.344513:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961517056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078495.344519:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078495.344522:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078495.344524:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961517056:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078495.855845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078496.049852:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078496.879844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078497.072063:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078497.904720:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078498.095994:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078498.927921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078499.120796:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078499.951871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078500.143958:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078500.464813:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078500.464823:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078500.464828:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078500.464838:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4183 00000100:00080000:2.0:1769078500.464848:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078500.464855:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078500.464891:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961517184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078500.464926:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078500.464942:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078500.464944:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078500.464948:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4183 00000100:00080000:2.0:1769078500.464952:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078500.465132:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078500.465138:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961517184:192.168.204.117@tcp:250:kworker.0 00000008:00100000:2.0:1769078500.465144:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078500.465163:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961517312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078500.465174:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078500.465177:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078500.465179:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078500.465182:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4183 00000100:00080000:2.0:1769078500.465190:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078500.465194:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000800:00000100:1.0:1769078500.465200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000008:00100000:2.0:1769078500.465200:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078500.465214:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961517440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078500.465254:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078500/real 1769078500] req@ffff895e44e29500 x1855008961517184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078500.465271:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961517184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078500.465279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078500.465282:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078500.465286:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961517184:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078500.465307:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961517312:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078500.465324:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078500.465334:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078500/real 1769078500] req@ffff895e44e28a80 x1855008961517312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078500.465343:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961517312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078500.465350:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078500.465353:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078500.465355:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961517312:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078500.465365:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961517440:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078500.465380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078500.465396:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078500/real 1769078500] req@ffff895e44e29180 x1855008961517440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078500.465406:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961517440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078500.465413:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078500.465416:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078500.465419:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961517440:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078500.975930:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078501.167969:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078501.999949:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078502.192364:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078503.023833:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078503.219234:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078504.047898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078504.239971:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078505.072774:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078505.268917:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078505.583907:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078505.583917:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078505.583921:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078505.583929:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4188 00000100:00080000:0.0:1769078505.583940:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078505.583945:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078505.583978:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961517568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078505.584003:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078505.584005:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078505.584006:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078505.584009:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4188 00000100:00080000:0.0:1769078505.584011:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078505.584013:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078505.584020:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078505.584031:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961517696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078505.584040:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078505.584056:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078505.584057:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078505.584061:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4188 00000100:00080000:0.0:1769078505.584065:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078505.584069:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078505.584074:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078505.584179:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961517824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078505.584251:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961517568:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078505.584313:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078505.584351:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078505/real 1769078505] req@ffff895e4383ea00 x1855008961517568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078505.584367:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961517568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078505.584377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078505.584380:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078505.584383:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961517568:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078505.584391:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961517696:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078505.584411:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078505.584430:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078505/real 1769078505] req@ffff895e4383dc00 x1855008961517696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078505.584443:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961517696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078505.584450:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078505.584453:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078505.584457:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961517696:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078505.584462:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961517824:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078505.584481:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078505.584490:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078505/real 1769078505] req@ffff895e4383ca80 x1855008961517824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078505.584501:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961517824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078505.584508:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078505.584511:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078505.584515:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961517824:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078506.095864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078506.288226:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078507.119877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078507.311921:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078508.145279:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078508.336049:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078509.167867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078509.359896:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078510.191844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078510.383864:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078510.703892:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078510.703900:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078510.703904:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078510.703912:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4193 00000100:00080000:3.0:1769078510.703921:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078510.703927:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078510.703970:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961517952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078510.703991:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078510.704009:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078510.704012:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078510.704016:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4193 00000100:00080000:3.0:1769078510.704021:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078510.704026:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078510.704040:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961517952:192.168.204.117@tcp:250:kworker.0 00000008:00100000:3.0:1769078510.704043:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078510.704072:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961518080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078510.704081:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078510.704084:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078510.704085:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078510.704089:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4193 00000100:00080000:3.0:1769078510.704093:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078510.704097:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078510.704102:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078510.704113:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961518208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078510.704117:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078510.704166:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078510/real 1769078510] req@ffff895e453d7800 x1855008961517952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078510.704182:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961517952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078510.704192:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078510.704196:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078510.704199:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961517952:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078510.704221:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961518080:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078510.704242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078510.704251:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078510/real 1769078510] req@ffff895e453d5c00 x1855008961518080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078510.704258:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961518080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078510.704264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078510.704266:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078510.704268:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961518080:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078510.704271:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961518208:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078510.704281:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078510.704286:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078510/real 1769078510] req@ffff895e453d5f80 x1855008961518208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078510.704293:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961518208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078510.704306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078510.704308:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078510.704310:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961518208:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078511.215872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078511.408024:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078512.239849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078512.431874:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078513.263858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078513.455951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078514.288033:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078514.479944:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078515.311888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078515.506053:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078515.824011:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078515.824024:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078515.824028:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078515.824038:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4198 00000100:00080000:0.0:1769078515.824049:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078515.824061:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078515.824097:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961518336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078515.824128:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078515.824131:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078515.824150:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078515.824155:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4198 00000100:00080000:0.0:1769078515.824159:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078515.824163:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078515.824172:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078515.824185:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961518464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078515.824195:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078515.824198:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078515.824199:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078515.824203:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4198 00000100:00080000:0.0:1769078515.824206:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078515.824209:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078515.824213:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078515.824223:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961518592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078515.824562:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961518336:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078515.824618:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078515.824665:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078515/real 1769078515] req@ffff895e4383d500 x1855008961518336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078515.824685:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961518336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078515.824695:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078515.824699:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078515.824703:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961518336:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078515.824710:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961518464:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078515.824733:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078515.824777:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078515/real 1769078515] req@ffff895e4383f800 x1855008961518464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078515.824788:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961518464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078515.824801:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078515.824803:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078515.824806:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961518464:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078515.824810:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961518592:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078515.824826:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078515.824833:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078515/real 1769078515] req@ffff895e4383e300 x1855008961518592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078515.824839:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961518592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078515.824844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078515.824846:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078515.824848:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961518592:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078516.336414:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078516.527923:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078517.359858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078517.551929:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078518.383873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078518.576002:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078519.408643:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078519.599932:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078520.431847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078520.624047:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1769078520.943942:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078520.943953:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078520.943958:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078520.943967:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4203 00000100:00080000:2.0:1769078520.943977:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078520.943984:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078520.944026:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961518720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078520.944060:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078520.944064:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078520.944066:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078520.944070:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4203 00000100:00080000:2.0:1769078520.944074:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078520.944077:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078520.944086:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078520.944098:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961518848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078520.944107:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078520.944111:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078520.944112:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078520.944116:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4203 00000100:00080000:2.0:1769078520.944133:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078520.944137:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078520.944142:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078520.944152:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961518976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078520.944239:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961518720:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078520.944295:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078520.944330:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078520/real 1769078520] req@ffff895e44e2bb80 x1855008961518720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078520.944347:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961518720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078520.944356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078520.944360:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078520.944370:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961518720:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078520.944377:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961518848:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078520.944401:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078520.944413:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078520/real 1769078520] req@ffff895e44e2b800 x1855008961518848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078520.944479:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961518848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078520.944490:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078520.944494:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078520.944497:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961518848:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078520.944504:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961518976:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078520.944524:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078520.944534:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078520/real 1769078520] req@ffff895e44e29c00 x1855008961518976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078520.944545:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961518976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078520.944552:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078520.944555:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078520.944558:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961518976:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078521.456216:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078521.647927:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078522.479870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078522.671930:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078523.503869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078523.695920:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078524.528271:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078524.721386:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078525.551872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078525.744037:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078526.063899:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078526.063918:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078526.063937:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078526.063946:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4208 00000100:00080000:0.0:1769078526.063956:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078526.063962:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078526.063994:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961519104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078526.064024:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078526.064027:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078526.064029:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078526.064033:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4208 00000100:00080000:0.0:1769078526.064036:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078526.064040:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078526.064048:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078526.064060:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961519232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078526.064077:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078526.064080:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078526.064082:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078526.064085:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4208 00000100:00080000:0.0:1769078526.064088:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078526.064091:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078526.064094:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078526.064104:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961519360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078526.064124:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961519104:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078526.064254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078526.064293:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078526/real 1769078526] req@ffff895e4383e300 x1855008961519104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078526.064309:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961519104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078526.064392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078526.064398:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078526.064402:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961519104:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078526.064410:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961519232:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078526.064433:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078526.064445:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078526/real 1769078526] req@ffff895e4383f800 x1855008961519232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078526.064458:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961519232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078526.064466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078526.064469:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078526.064472:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961519232:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078526.064495:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961519360:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078526.064514:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078526.064525:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078526/real 1769078526] req@ffff895e4383d500 x1855008961519360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078526.064536:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961519360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078526.064553:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078526.064557:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078526.064560:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961519360:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078526.575876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078526.770955:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078527.599856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078527.792064:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078528.624222:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078528.816999:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078529.647923:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078529.840264:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078530.672849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078530.863928:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078531.184068:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078531.184078:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078531.184082:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078531.184091:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4213 00000100:00080000:1.0:1769078531.184102:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078531.184109:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078531.184139:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961519488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078531.184162:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078531.184165:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078531.184167:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078531.184171:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4213 00000100:00080000:1.0:1769078531.184174:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078531.184178:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078531.184186:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078531.184191:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961519488:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078531.184196:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961519616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078531.184205:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078531.184221:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078531.184223:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078531.184226:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4213 00000100:00080000:1.0:1769078531.184229:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078531.184232:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078531.184236:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078531.184244:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961519744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078531.184276:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961519616:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078531.184301:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961519744:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078531.695924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078531.888485:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078532.720472:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078532.914535:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078533.744774:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078533.935919:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1769078534.258611:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769078534.259199:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078534.259217:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078534.259259:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078534.259288:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:1.0:1769078534.259409:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078531/real 1769078534] req@ffff895e4713ce00 x1855008961519616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078534.259436:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961519616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078534.259448:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078534.259453:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078534.259458:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961519616:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078534.259466:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078531/real 1769078534] req@ffff895e4713d500 x1855008961519744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078534.259484:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961519744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078534.259493:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078534.259496:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078534.259505:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961519744:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078534.259512:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078531/real 1769078534] req@ffff895e4713f480 x1855008961519488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078534.259526:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961519488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078534.259534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078534.259538:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078534.259541:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961519488:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078534.768012:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078534.961115:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078535.792058:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769078535.985789:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078536.304882:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078536.304897:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078536.304902:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078536.304914:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4218 00000100:00080000:1.0:1769078536.304929:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078536.304937:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078536.304982:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961519872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078536.305023:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078536.305027:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078536.305029:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078536.305033:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4218 00000100:00080000:1.0:1769078536.305037:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078536.305042:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078536.305053:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078536.305087:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961520000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078536.305098:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078536.305102:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078536.305105:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078536.305108:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4218 00000100:00080000:1.0:1769078536.305113:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078536.305117:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078536.305122:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078536.305132:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961520128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078536.306954:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961519872:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769078536.307313:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078536.307390:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078536/real 1769078536] req@ffff895e4713f480 x1855008961519872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078536.307412:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961519872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078536.307424:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078536.307429:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078536.307434:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961519872:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078536.307443:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961520000:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078536.307479:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078536.307493:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078536/real 1769078536] req@ffff895e4713d500 x1855008961520000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078536.307504:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961520000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078536.307513:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078536.307517:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078536.307520:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961520000:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078536.307528:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961520128:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078536.310235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078536.310280:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078536/real 1769078536] req@ffff895e4713ce00 x1855008961520128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078536.310304:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961520128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078536.310315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078536.310320:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078536.310323:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961520128:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078536.816241:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078537.007983:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078537.839830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078538.032093:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078538.863890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078539.055949:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078539.887822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078540.079956:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078540.912151:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078541.124941:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078541.423958:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078541.423972:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078541.423978:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078541.423990:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4223 00000100:00080000:0.0:1769078541.424003:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078541.424012:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078541.424056:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961520256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078541.424084:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078541.424088:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078541.424090:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078541.424094:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4223 00000100:00080000:0.0:1769078541.424097:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078541.424101:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078541.424112:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078541.424126:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961520384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078541.424149:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961520256:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769078541.424165:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078541.424168:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078541.424170:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078541.424175:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4223 00000100:00080000:0.0:1769078541.424179:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078541.424183:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078541.424189:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078541.424201:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961520512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078541.424243:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078541.424294:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078541/real 1769078541] req@ffff895e4383ca80 x1855008961520256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078541.424313:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961520256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078541.424325:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078541.424329:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078541.424333:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961520256:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078541.424359:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961520384:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078541.424392:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078541.424404:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078541/real 1769078541] req@ffff895e4383dc00 x1855008961520384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078541.424417:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961520384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078541.424426:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078541.424429:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078541.424432:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961520384:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078541.424438:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961520512:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078541.424459:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078541.424470:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078541/real 1769078541] req@ffff895e4383ea00 x1855008961520512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078541.424478:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961520512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078541.424485:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078541.424488:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078541.424491:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961520512:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078541.935899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078542.127981:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078542.959892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078543.152772:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078543.983904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078544.175941:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078545.007919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078545.199906:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1769078546.031902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078546.224444:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078546.544425:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078546.544437:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078546.544441:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078546.544458:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4229 00000100:00080000:0.0:1769078546.544469:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078546.544476:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078546.544509:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961520640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078546.544537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078546.544540:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078546.544542:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078546.544546:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4229 00000100:00080000:0.0:1769078546.544549:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078546.544569:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078546.544580:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078546.544592:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961520640:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078546.544595:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961520768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078546.544605:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078546.544608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078546.544610:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078546.544612:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4229 00000100:00080000:0.0:1769078546.544615:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078546.544618:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078546.544621:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078546.544628:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961520896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078546.544658:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078546.544697:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078546/real 1769078546] req@ffff895e4383c000 x1855008961520640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078546.544715:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961520640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078546.544726:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078546.544730:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078546.544734:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961520640:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078546.544784:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961520768:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078546.544814:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078546.544825:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078546/real 1769078546] req@ffff895e4383f480 x1855008961520768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078546.544837:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961520768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078546.544846:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078546.544849:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078546.544853:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961520768:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078546.544859:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961520896:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078546.544876:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078546.544885:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078546/real 1769078546] req@ffff895e4383ce00 x1855008961520896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078546.544908:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961520896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078546.544916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078546.544919:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078546.544922:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961520896:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078547.055860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078547.250495:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078548.079902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078548.271937:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078549.103860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078549.295939:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078550.127866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078550.319898:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078551.151822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078551.343910:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078551.663861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078551.663871:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078551.663875:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078551.663886:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4234 00000100:00080000:0.0:1769078551.663895:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078551.663903:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078551.663938:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961521024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078551.663964:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078551.663981:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078551.663982:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078551.663986:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4234 00000100:00080000:0.0:1769078551.663990:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078551.663994:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078551.664002:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078551.664015:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961521024:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078551.664016:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961521152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078551.664037:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078551.664041:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078551.664043:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078551.664047:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4234 00000100:00080000:0.0:1769078551.664050:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078551.664054:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078551.664058:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078551.664067:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961521280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078551.664086:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078551.664126:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078551/real 1769078551] req@ffff895e4383fb80 x1855008961521024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078551.664153:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961521024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078551.664161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078551.664165:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078551.664169:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961521024:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078551.664190:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961521152:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078551.664211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078551.664222:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078551/real 1769078551] req@ffff895e4383c380 x1855008961521152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078551.664234:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961521152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078551.664252:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078551.664256:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078551.664259:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961521152:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078551.664265:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961521280:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078551.664285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078551.664295:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078551/real 1769078551] req@ffff895e4383f100 x1855008961521280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078551.664307:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961521280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078551.664315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078551.664318:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078551.664321:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961521280:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078552.175865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078552.367906:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078553.199821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078553.391897:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078554.223821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078554.415877:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078555.247848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078555.440002:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078556.271875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078556.463971:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769078556.784342:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078556.784351:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078556.784355:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078556.784364:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4239 00000100:00080000:1.0:1769078556.784392:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078556.784398:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078556.784429:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961521408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078556.784464:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078556.784468:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078556.784470:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078556.784473:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4239 00000100:00080000:1.0:1769078556.784477:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078556.784481:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078556.784489:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078556.784501:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961521536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078556.784511:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078556.784514:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078556.784516:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078556.784519:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4239 00000100:00080000:1.0:1769078556.784523:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078556.784526:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078556.784530:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078556.784540:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961521664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078556.784640:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961521408:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769078556.784694:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078556.784727:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078556/real 1769078556] req@ffff895e4713ea00 x1855008961521408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078556.784742:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961521408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078556.784777:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078556.784781:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078556.784790:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961521408:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078556.784797:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961521536:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078556.784819:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078556.784830:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078556/real 1769078556] req@ffff895e4713f100 x1855008961521536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078556.784841:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961521536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078556.784850:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078556.784852:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078556.784856:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961521536:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078556.784861:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961521664:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078556.784876:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078556.784884:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078556/real 1769078556] req@ffff895e4713e300 x1855008961521664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078556.784895:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961521664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078556.784902:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078556.784905:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078556.784908:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961521664:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078557.295842:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078557.487940:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078558.319869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078558.511896:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078559.343853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078559.535917:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078560.367838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078560.562834:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078561.391838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078561.585255:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078561.903850:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078561.903860:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078561.903864:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078561.903871:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4244 00000100:00080000:3.0:1769078561.903879:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078561.903884:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078561.903915:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961521792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078561.903937:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078561.903939:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078561.903941:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078561.903943:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4244 00000100:00080000:3.0:1769078561.903945:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078561.903947:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078561.903954:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078561.903971:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961521920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078561.903983:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961521792:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078561.903983:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078561.903992:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078561.904009:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078561.904014:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4244 00000100:00080000:3.0:1769078561.904019:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078561.904024:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078561.904029:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078561.904039:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961522048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769078561.904050:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078561.904088:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078561/real 1769078561] req@ffff895e453d7480 x1855008961521792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078561.904107:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961521792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078561.904114:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078561.904117:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078561.904120:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961521792:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078561.904139:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961521920:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078561.904153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078561.904160:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078561/real 1769078561] req@ffff895e453d4a80 x1855008961521920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078561.904168:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961521920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078561.904173:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078561.904175:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078561.904177:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961521920:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078561.904180:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961522048:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078561.904204:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078561.904219:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078561/real 1769078561] req@ffff895e453d4e00 x1855008961522048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078561.904226:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961522048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078561.904240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078561.904242:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078561.904244:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961522048:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078562.415832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078562.607885:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078563.439856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078563.631891:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078564.463823:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078564.655899:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078565.488214:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078565.681196:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078566.514725:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078566.703964:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078567.023858:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078567.023868:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078567.023872:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078567.023883:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4249 00000100:00080000:3.0:1769078567.024031:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078567.024040:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078567.024074:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961522176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078567.024121:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078567.024125:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078567.024127:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078567.024132:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4249 00000100:00080000:3.0:1769078567.024136:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078567.024140:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078567.024165:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078567.024178:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961522304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078567.024189:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078567.024192:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078567.024194:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078567.024197:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4249 00000100:00080000:3.0:1769078567.024202:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078567.024205:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078567.024210:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078567.024219:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961522432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078567.024376:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961522176:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769078567.024436:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078567.024472:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078567/real 1769078567] req@ffff895e453d5500 x1855008961522176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078567.024489:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961522176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078567.024498:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078567.024502:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078567.024505:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961522176:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078567.024512:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961522304:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078567.024532:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078567.024544:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078567/real 1769078567] req@ffff895e453d7b80 x1855008961522304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078567.024556:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961522304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078567.024564:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078567.024567:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078567.024570:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961522304:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078567.024584:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961522432:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078567.024604:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078567.024613:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078567/real 1769078567] req@ffff895e453d7100 x1855008961522432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078567.024625:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961522432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078567.024632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078567.024635:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078567.024638:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961522432:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078567.535869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078567.727925:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078568.559869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078568.752010:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078569.610579:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078569.777564:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078570.674172:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078570.800035:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078571.696826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078571.823929:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769078572.144008:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078572.145066:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078572.145072:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078572.145086:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4254 00000100:00080000:3.0:1769078572.145189:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078572.145200:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078572.145259:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961522560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078572.145295:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078572.145299:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078572.145300:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078572.145304:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4254 00000100:00080000:3.0:1769078572.145328:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078572.145333:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078572.145343:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078572.145354:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961522688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078572.145364:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078572.145367:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078572.145368:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078572.145371:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4254 00000100:00080000:3.0:1769078572.145374:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078572.145378:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078572.145382:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078572.145389:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961522816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078572.145607:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961522560:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769078572.145690:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078572.145732:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078572/real 1769078572] req@ffff895e453d4700 x1855008961522560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078572.145788:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961522560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078572.145799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078572.145804:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078572.145807:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961522560:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078572.145815:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961522688:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078572.145837:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078572.145849:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078572/real 1769078572] req@ffff895e453d5880 x1855008961522688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078572.145877:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961522688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078572.145886:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078572.145889:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078572.145892:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961522688:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078572.145898:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961522816:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078572.145920:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078572.146148:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078572/real 1769078572] req@ffff895e453d5180 x1855008961522816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078572.146161:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961522816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078572.146168:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078572.146171:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078572.146174:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961522816:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078572.719950:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078572.848078:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078573.743931:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078573.872568:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078574.767912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078574.896430:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078575.792763:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078575.920011:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078576.816039:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078576.944101:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078577.263946:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078577.264033:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078577.264038:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078577.264048:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4259 00000100:00080000:0.0:1769078577.264059:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078577.264066:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078577.264102:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961522944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078577.264133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078577.264136:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078577.264138:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078577.264142:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4259 00000100:00080000:0.0:1769078577.264155:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078577.264160:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078577.264169:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078577.264181:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961523072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078577.264190:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078577.264193:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078577.264195:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078577.264199:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4259 00000100:00080000:0.0:1769078577.264202:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078577.264205:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078577.264210:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078577.264235:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961523200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078577.264625:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961522944:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769078577.264827:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078577.264863:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078577/real 1769078577] req@ffff895e4383c700 x1855008961522944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078577.264880:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961522944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078577.264904:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078577.264908:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078577.264912:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961522944:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078577.264919:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961523072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078577.265039:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078577.265054:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078577/real 1769078577] req@ffff895e4383e680 x1855008961523072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078577.265066:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961523072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078577.265074:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078577.265077:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078577.265080:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961523072:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078577.265086:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961523200:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078577.265104:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078577.265114:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078577/real 1769078577] req@ffff895e4383d880 x1855008961523200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078577.265124:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961523200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078577.265130:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078577.265132:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078577.265135:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961523200:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078577.839854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078577.968360:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078578.864073:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078578.992816:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078579.889822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078580.016101:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078580.911862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078581.040109:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078581.936239:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078582.073458:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769078582.383890:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078582.383902:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078582.383907:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078582.383916:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4264 00000100:00080000:0.0:1769078582.383926:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078582.383932:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078582.383969:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961523328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078582.384005:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078582.384009:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078582.384011:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078582.384015:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4264 00000100:00080000:0.0:1769078582.384019:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078582.384022:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078582.384031:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078582.384042:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961523456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078582.384052:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078582.384055:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078582.384056:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078582.384072:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4264 00000100:00080000:0.0:1769078582.384076:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078582.384080:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078582.384085:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078582.384097:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961523584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078582.384245:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961523328:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769078582.384304:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078582.384343:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078582/real 1769078582] req@ffff895e4383d180 x1855008961523328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078582.384360:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961523328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078582.384369:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078582.384372:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078582.384376:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961523328:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078582.384382:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961523456:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078582.384403:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078582.384414:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078582/real 1769078582] req@ffff895e4383df80 x1855008961523456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078582.384425:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961523456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078582.384433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078582.388688:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078582.388698:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961523456:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078582.389517:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961523584:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078582.389587:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078582.389614:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078582/real 1769078582] req@ffff895e4383ed80 x1855008961523584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078582.389630:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961523584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078582.389640:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078582.389644:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078582.389647:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961523584:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078582.960015:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078583.512916:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078583.984200:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078584.560120:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078585.007874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078585.583973:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078586.031916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078586.609097:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078587.055881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078587.505577:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078587.505600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078587.505604:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078587.505614:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4270 00000100:00080000:3.0:1769078587.505818:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078587.505827:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078587.505862:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961523712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078587.505882:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078587.505885:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078587.505887:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078587.505891:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4270 00000100:00080000:3.0:1769078587.505894:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078587.505898:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078587.505908:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078587.505921:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961523840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078587.505930:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078587.505933:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078587.505935:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078587.505938:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4270 00000100:00080000:3.0:1769078587.505941:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078587.505945:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078587.505949:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078587.505959:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961523968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078587.506395:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961523712:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078587.506462:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078587.506500:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078587/real 1769078587] req@ffff895e453d4380 x1855008961523712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078587.506517:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961523712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078587.506534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078587.506539:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078587.506543:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961523712:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078587.506550:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961523840:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078587.506574:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078587.506587:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078587/real 1769078587] req@ffff895e453d6680 x1855008961523840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078587.506802:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961523840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078587.506810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078587.506813:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078587.506816:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961523840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078587.506822:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961523968:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078587.506843:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078587.506855:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078587/real 1769078587] req@ffff895e453d6300 x1855008961523968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078587.506866:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961523968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078587.506874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078587.506877:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078587.506881:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961523968:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078587.751389:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078588.080780:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078588.720164:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078589.104079:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078589.743950:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078590.127899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078590.769331:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078591.151880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078591.792089:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078592.175873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078592.669298:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078592.669308:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078592.669312:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078592.669321:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4275 00000100:00080000:3.0:1769078592.669332:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078592.669340:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078592.669377:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961524096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078592.669411:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078592.669415:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078592.669417:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078592.669421:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4275 00000100:00080000:3.0:1769078592.669424:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078592.669428:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078592.669437:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078592.669451:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961524224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078592.669461:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078592.669464:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078592.669466:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078592.669470:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4275 00000100:00080000:3.0:1769078592.669487:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078592.669491:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078592.669496:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078592.669507:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961524352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078592.702143:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961524096:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078592.702223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078592.702284:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078592/real 1769078592] req@ffff895e453d6d80 x1855008961524096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078592.702304:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961524096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078592.702315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078592.702319:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078592.702323:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961524096:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078592.702332:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961524224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078592.702355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078592.702368:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078592/real 1769078592] req@ffff895e453d6300 x1855008961524224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078592.702380:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961524224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078592.702388:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078592.702391:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078592.702394:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961524224:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078592.702400:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961524352:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078592.702437:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078592.702448:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078592/real 1769078592] req@ffff895e453d6680 x1855008961524352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078592.702460:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961524352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078592.702466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078592.702470:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078592.702473:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961524352:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078592.816046:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078593.200187:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078593.843037:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078594.224098:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078594.864040:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078595.271784:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078595.889131:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078596.271912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078596.913542:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078597.317376:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078597.743891:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078597.743901:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078597.743906:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078597.743917:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4280 00000100:00080000:3.0:1769078597.743928:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078597.743936:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078597.743971:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961524480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078597.744007:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078597.744010:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078597.744012:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078597.744016:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4280 00000100:00080000:3.0:1769078597.744020:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078597.744024:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078597.744034:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078597.744064:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961524608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078597.744087:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078597.744090:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078597.744092:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078597.744096:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4280 00000100:00080000:3.0:1769078597.744100:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078597.744104:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078597.744108:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078597.744118:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961524736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078597.744257:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961524480:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078597.744362:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961524608:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078597.744389:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961524736:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078597.936042:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078598.319869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078598.960145:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078599.343804:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078599.985264:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078600.367863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769078600.816075:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769078600.816122:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078600.816139:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078600.816198:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078600.816206:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769078600.820291:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078597/real 1769078600] req@ffff895e453d4380 x1855008961524480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078600.820325:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961524480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078600.820337:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078600.820359:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078600.820364:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961524480:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078600.820372:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078597/real 1769078600] req@ffff895e453d5180 x1855008961524608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078600.820386:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961524608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078600.820407:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078600.820410:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078600.820414:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961524608:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078600.820420:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078597/real 1769078600] req@ffff895e453d5880 x1855008961524736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078600.820431:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961524736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078600.820439:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078600.820442:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078600.820445:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961524736:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078601.008027:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078601.395024:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078602.033552:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078602.415964:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078602.893622:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078602.893634:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078602.893638:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078602.893647:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4285 00000100:00080000:1.0:1769078602.893658:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078602.893665:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078602.894273:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961524864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078602.894512:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961524864:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078602.894583:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:1.0:1769078602.894693:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078602.894706:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078602.894709:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078602.894714:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4285 00000100:00080000:1.0:1769078602.894718:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078602.894723:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078602.894732:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078602.894765:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961524992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078602.895373:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078602/real 1769078602] req@ffff895e4713d880 x1855008961524864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078602.895427:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961524864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078602.895436:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078602.895441:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078602.895445:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961524864:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078602.895687:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961524992:192.168.204.117@tcp:38:kworker.0 00000100:00080000:1.0:1769078602.901033:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078602.901044:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078602.901048:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078602.901077:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4285 00000100:00080000:1.0:1769078602.901089:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078602.901096:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078602.901109:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078602.901142:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961525120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078602.902270:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078602.903277:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078602/real 1769078602] req@ffff895e4713ed80 x1855008961524992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078602.903627:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961524992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078602.903638:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078602.903643:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078602.903647:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961524992:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078602.903679:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961525120:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078602.904673:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078602.904696:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078602/real 1769078602] req@ffff895e4713c000 x1855008961525120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078602.905103:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961525120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078602.905134:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078602.905139:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078602.905143:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961525120:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078603.057694:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078603.439944:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078604.080258:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078604.463877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078605.104823:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078605.487956:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078606.127946:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078606.511849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078607.152048:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078607.535891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078607.983938:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078607.983949:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078607.983953:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078607.983963:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4290 00000100:00080000:3.0:1769078607.983973:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078607.983980:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078607.984020:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961525248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078607.984090:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961525248:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078607.984147:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769078607.984176:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078607.984180:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078607.984195:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078607.984199:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4290 00000100:00080000:3.0:1769078607.984203:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078607.984207:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078607.984216:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078607.984229:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961525376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078607.984283:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078607/real 1769078607] req@ffff895e453d4700 x1855008961525248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078607.984300:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961525248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078607.984309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078607.984313:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078607.984317:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961525248:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078607.984340:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961525376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078607.984362:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078607.984372:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078607/real 1769078607] req@ffff895e453d7100 x1855008961525376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078607.984383:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961525376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078607.984403:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078607.984406:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078607.984409:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961525376:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769078607.984506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078607.984510:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078607.984512:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078607.984516:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4290 00000100:00080000:3.0:1769078607.984520:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078607.984524:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078607.984529:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078607.984540:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961525504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078607.984579:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961525504:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078607.984600:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078607.984660:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078607/real 1769078607] req@ffff895e453d7b80 x1855008961525504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078607.984672:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961525504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078607.984679:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078607.984682:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078607.984685:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961525504:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078608.176035:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078608.559904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078609.200692:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078609.583885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078610.224062:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078610.607858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078611.247942:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078611.632156:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078612.315169:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078612.655880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078613.104987:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078613.104999:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078613.105004:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078613.105013:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4295 00000100:00080000:2.0:1769078613.105024:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078613.105031:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078613.105068:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961525632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078613.105106:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078613.105110:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078613.105112:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078613.105116:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4295 00000100:00080000:2.0:1769078613.105120:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078613.105123:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078613.105132:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078613.105144:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961525760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078613.105154:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078613.105157:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078613.105158:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078613.105161:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4295 00000100:00080000:2.0:1769078613.105164:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078613.105184:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078613.105189:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078613.105200:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961525888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078613.106436:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961525632:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078613.106501:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078613.106539:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078613/real 1769078613] req@ffff895e44e29c00 x1855008961525632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078613.106557:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961525632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078613.106583:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078613.106587:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078613.106591:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961525632:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078613.106598:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961525760:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078613.106619:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078613.106629:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078613/real 1769078613] req@ffff895e44e2b800 x1855008961525760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078613.106638:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961525760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078613.106645:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078613.106647:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078613.106650:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961525760:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078613.106654:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961525888:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078613.106666:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078613.106673:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078613/real 1769078613] req@ffff895e44e2bb80 x1855008961525888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078613.106681:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961525888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078613.106686:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078613.106689:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078613.106691:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961525888:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078613.295924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078613.679882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078614.319969:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078614.706939:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078615.350990:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078615.729415:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078616.368694:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078616.751905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078617.391927:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078617.777023:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078618.223934:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078618.223945:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078618.223949:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078618.223958:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4300 00000100:00080000:3.0:1769078618.223967:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078618.223974:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078618.224022:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961526016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078618.224070:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078618.224074:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078618.224076:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078618.224080:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4300 00000100:00080000:3.0:1769078618.224085:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078618.224088:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078618.224097:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078618.224108:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961526144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078618.224117:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078618.224120:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078618.224122:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078618.224125:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4300 00000100:00080000:3.0:1769078618.224128:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078618.224131:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078618.224135:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078618.224145:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961526272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078618.224295:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961526016:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078618.224355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078618.224389:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078618/real 1769078618] req@ffff895e453d5500 x1855008961526016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078618.224406:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961526016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078618.224415:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078618.224419:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078618.224423:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961526016:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078618.224430:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961526144:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078618.224450:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078618.224460:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078618/real 1769078618] req@ffff895e453d4e00 x1855008961526144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078618.224472:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961526144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078618.224480:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078618.224484:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078618.224487:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961526144:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078618.224493:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961526272:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078618.224520:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078618.224530:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078618/real 1769078618] req@ffff895e453d4a80 x1855008961526272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078618.224541:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961526272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078618.224549:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078618.224552:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078618.224555:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961526272:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078618.418313:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078618.802486:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078619.440076:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078619.823929:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078620.470815:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078620.849138:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078621.489128:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078621.872352:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078622.512090:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078622.896927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078623.345469:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078623.345481:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078623.345485:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078623.345509:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4305 00000100:00080000:3.0:1769078623.345540:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078623.345549:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078623.345588:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961526400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078623.345625:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078623.345628:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078623.345630:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078623.345634:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4305 00000100:00080000:3.0:1769078623.345637:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078623.345641:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078623.345650:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078623.345661:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961526528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078623.345671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078623.345673:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078623.345675:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078623.345679:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4305 00000100:00080000:3.0:1769078623.345681:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078623.345685:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078623.345689:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078623.345696:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961526656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078623.347342:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961526400:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078623.347421:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078623.347463:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078623/real 1769078623] req@ffff895e453d7480 x1855008961526400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078623.347481:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961526400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078623.347491:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078623.347509:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078623.347512:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961526400:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078623.347520:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961526528:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078623.347541:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078623.347568:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078623/real 1769078623] req@ffff895e453d5f80 x1855008961526528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078623.347579:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961526528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078623.347587:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078623.347591:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078623.347594:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961526528:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078623.347600:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961526656:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078623.347618:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078623.347628:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078623/real 1769078623] req@ffff895e453d5c00 x1855008961526656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078623.347639:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961526656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078623.347645:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078623.347649:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078623.347651:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961526656:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078623.536659:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078623.919884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078624.560068:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078624.945580:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078625.584083:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078625.968961:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078626.608137:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078626.991887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078627.633308:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078628.016094:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078628.465261:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078628.465286:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078628.465291:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078628.465300:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4311 00000100:00080000:3.0:1769078628.465311:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078628.465319:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078628.465353:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961526784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078628.465388:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078628.465392:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078628.465394:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078628.465397:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4311 00000100:00080000:3.0:1769078628.465401:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078628.465405:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078628.465413:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078628.465427:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961526912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078628.465435:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078628.465438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078628.465440:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078628.465443:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4311 00000100:00080000:3.0:1769078628.465446:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078628.465449:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078628.465454:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078628.465463:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961527040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078628.468260:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961526784:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078628.468335:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078628.468391:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078628/real 1769078628] req@ffff895e453d7800 x1855008961526784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078628.468411:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961526784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078628.468422:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078628.468426:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078628.468430:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961526784:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078628.468437:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961526912:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078628.468460:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078628.468471:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078628/real 1769078628] req@ffff895e453d5c00 x1855008961526912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078628.468482:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961526912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078628.468490:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078628.468493:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078628.468496:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961526912:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078628.468502:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961527040:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078628.468521:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078628.468530:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078628/real 1769078628] req@ffff895e453d5f80 x1855008961527040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078628.468540:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961527040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078628.468558:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078628.468561:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078628.468565:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961527040:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078628.656261:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078629.040010:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078629.680792:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078630.063862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078630.705225:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078631.089484:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078631.727986:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078632.111907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078632.751973:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078633.135897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078633.583837:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078633.583857:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078633.583861:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078633.583876:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4316 00000100:00080000:3.0:1769078633.583887:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078633.583893:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078633.583928:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961527168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078633.584005:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961527168:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078633.584085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769078633.584115:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078633.584119:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078633.584121:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078633.584125:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4316 00000100:00080000:3.0:1769078633.584129:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078633.584133:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078633.584143:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078633.584155:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961527296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078633.584807:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078633/real 1769078633] req@ffff895e453d7480 x1855008961527168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078633.584828:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961527168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078633.584837:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078633.586434:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078633.586442:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961527168:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078633.586478:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961527296:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078633.586543:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078633.586579:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078633/real 1769078633] req@ffff895e453d4a80 x1855008961527296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078633.586598:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961527296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078633.586632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078633.586636:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078633.586639:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961527296:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769078633.586822:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078633.586827:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078633.586830:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078633.586834:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4316 00000100:00080000:3.0:1769078633.586842:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078633.586860:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078633.586889:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078633.586913:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961527424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078633.587085:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961527424:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078633.587110:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078633.587122:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078633/real 1769078633] req@ffff895e453d4e00 x1855008961527424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078633.587133:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961527424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078633.587141:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078633.587145:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078633.587148:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961527424:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078633.789023:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078634.159882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078634.801567:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078635.183905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078635.825983:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078636.207902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078636.848157:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078637.231950:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078637.872196:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078638.256128:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078638.704621:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078638.704641:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078638.704645:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078638.704655:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4321 00000100:00080000:3.0:1769078638.704668:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078638.704675:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078638.704710:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961527552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078638.704742:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078638.704764:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078638.704766:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078638.704771:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4321 00000100:00080000:3.0:1769078638.704774:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078638.704778:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078638.704788:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078638.704799:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961527680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078638.704808:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078638.704811:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078638.704812:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078638.704816:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4321 00000100:00080000:3.0:1769078638.704833:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078638.704848:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078638.704854:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078638.704864:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961527808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078638.704985:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961527552:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078638.705072:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078638.705110:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078638/real 1769078638] req@ffff895e453d5500 x1855008961527552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078638.705135:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961527552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078638.705157:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078638.705161:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078638.705165:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961527552:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078638.705173:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961527680:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078638.705198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078638.705211:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078638/real 1769078638] req@ffff895e453d7b80 x1855008961527680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078638.705224:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961527680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078638.705232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078638.705235:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078638.705238:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961527680:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078638.705243:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961527808:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078638.705261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078638.705271:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078638/real 1769078638] req@ffff895e453d7100 x1855008961527808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078638.705281:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961527808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078638.705289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078638.705292:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078638.705295:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961527808:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078638.896715:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078639.279870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078639.920017:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078640.305568:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078640.944113:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078641.328559:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078641.968166:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078642.351987:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078642.992229:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078643.375971:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078643.826093:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078643.826105:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078643.826110:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078643.826121:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4326 00000100:00080000:3.0:1769078643.826133:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078643.826139:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078643.826178:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961527936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078643.826210:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078643.826213:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078643.826215:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078643.826219:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4326 00000100:00080000:3.0:1769078643.826223:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078643.826226:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078643.826235:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078643.826249:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961528064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078643.826257:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078643.826276:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078643.826278:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078643.826282:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4326 00000100:00080000:3.0:1769078643.826286:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078643.826290:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078643.826295:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078643.826306:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961528192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078643.826402:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961527936:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078643.826464:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078643.826499:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078643/real 1769078643] req@ffff895e453d4700 x1855008961527936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078643.826516:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961527936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078643.826524:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078643.826528:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078643.826532:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961527936:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078643.826538:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961528064:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078643.826559:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078643.826570:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078643/real 1769078643] req@ffff895e453d5880 x1855008961528064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078643.826581:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961528064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078643.826587:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078643.826590:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078643.826592:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961528064:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078643.826606:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961528192:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078643.826625:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078643.826634:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078643/real 1769078643] req@ffff895e453d5180 x1855008961528192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078643.826644:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961528192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078643.826651:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078643.826653:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078643.826656:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961528192:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078644.016295:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078644.400636:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078645.039958:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078645.424101:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078646.064009:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078646.447915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078647.087973:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078647.472838:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078648.112474:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078648.495888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078648.943990:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078648.944000:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078648.944004:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078648.944014:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4331 00000100:00080000:2.0:1769078648.944026:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078648.944034:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078648.944072:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961528320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078648.944102:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078648.944106:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078648.944108:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078648.944112:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4331 00000100:00080000:2.0:1769078648.944116:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078648.944119:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078648.944128:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078648.944139:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961528448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078648.944160:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078648.944163:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078648.944178:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078648.944447:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4331 00000100:00080000:2.0:1769078648.944452:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078648.944456:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078648.944461:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078648.944474:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961528576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078648.944622:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961528320:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078648.944682:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078648.944736:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078648/real 1769078648] req@ffff895e44e29180 x1855008961528320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078648.944779:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961528320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078648.944790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078648.944793:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078648.944797:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961528320:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078648.944804:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961528448:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078648.944846:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078648.944861:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078648/real 1769078648] req@ffff895e44e28a80 x1855008961528448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078648.944885:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961528448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078648.944893:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078648.944897:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078648.944900:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961528448:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078648.944905:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961528576:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078648.944925:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078648.944936:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078648/real 1769078648] req@ffff895e44e29500 x1855008961528576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078648.944946:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961528576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078648.944954:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078648.944956:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078648.944959:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961528576:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078649.143157:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078649.519954:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078650.159978:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078650.543939:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078651.183976:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078651.567872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078652.208580:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078652.593374:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078653.232441:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078653.616280:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078654.067090:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078654.067101:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078654.067106:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078654.067116:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4336 00000100:00080000:3.0:1769078654.067128:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078654.067135:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078654.067171:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961528704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078654.067203:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078654.067206:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078654.067208:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078654.067212:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4336 00000100:00080000:3.0:1769078654.067216:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078654.067220:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078654.067229:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078654.067241:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961528832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078654.067264:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078654.067267:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078654.067268:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078654.067272:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4336 00000100:00080000:3.0:1769078654.067275:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078654.067279:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078654.067284:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078654.067294:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961528960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078654.067551:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961528704:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078654.067607:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078654.067665:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078654/real 1769078654] req@ffff895e453d4380 x1855008961528704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078654.067681:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961528704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078654.067690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078654.067701:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078654.067705:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961528704:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078654.067713:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961528832:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078654.067735:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078654.067783:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078654/real 1769078654] req@ffff895e453d6680 x1855008961528832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078654.067796:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961528832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078654.067803:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078654.067806:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078654.067809:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961528832:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078654.067815:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961528960:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078654.067834:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078654.067844:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078654/real 1769078654] req@ffff895e453d6300 x1855008961528960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078654.067855:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961528960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078654.067874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078654.067877:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078654.067881:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961528960:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078654.255939:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078654.643498:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078655.284272:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078655.663952:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078656.304043:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078656.687912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078657.328678:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078657.711895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078658.355730:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078658.736684:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078659.183927:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078659.183940:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078659.183945:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078659.183955:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4341 00000100:00080000:3.0:1769078659.183965:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078659.183973:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078659.184011:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961529088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078659.184043:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078659.184051:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078659.184053:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078659.184057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4341 00000100:00080000:3.0:1769078659.184061:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078659.184073:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078659.184097:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078659.184111:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961529216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078659.184121:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078659.184124:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078659.184125:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078659.184129:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4341 00000100:00080000:3.0:1769078659.184132:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078659.184136:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078659.184141:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078659.184149:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961529344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078659.185023:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961529088:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078659.185099:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078659.185147:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078659/real 1769078659] req@ffff895e453d6d80 x1855008961529088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078659.185165:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961529088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078659.185175:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078659.185179:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078659.185183:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961529088:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078659.185189:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961529216:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078659.185220:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078659.185233:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078659/real 1769078659] req@ffff895e453d6300 x1855008961529216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078659.185242:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961529216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078659.185250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078659.185252:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078659.185255:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961529216:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078659.185260:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961529344:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078659.185275:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078659.185283:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078659/real 1769078659] req@ffff895e453d6680 x1855008961529344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078659.185293:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961529344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078659.185300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078659.185303:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078659.185306:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961529344:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078659.376060:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078659.759896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078660.406066:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078660.786460:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078661.424524:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078661.807869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078662.447961:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078662.831866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078663.472771:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078663.856087:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078664.304097:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078664.304108:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078664.304112:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078664.304122:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4346 00000100:00080000:2.0:1769078664.304133:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078664.304140:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078664.304174:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961529472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078664.304204:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078664.304207:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078664.304209:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078664.304213:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4346 00000100:00080000:2.0:1769078664.304216:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078664.304220:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078664.304228:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078664.304239:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961529600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078664.304268:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078664.304271:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078664.304273:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078664.304276:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4346 00000100:00080000:2.0:1769078664.304279:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078664.304283:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078664.304287:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078664.304296:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961529728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078664.304375:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961529472:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078664.304478:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961529600:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078664.304508:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961529728:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078664.495984:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078664.880585:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078665.520087:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078665.903868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078666.546785:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078666.927872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769078667.376839:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769078667.376891:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078667.376910:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078667.376958:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078667.376968:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:1.0:1769078667.377860:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078664/real 1769078667] req@ffff895e44e29f80 x1855008961529472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078667.377895:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961529472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078667.377907:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078667.377912:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078667.377917:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961529472:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078667.377935:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078664/real 1769078667] req@ffff895e44e28e00 x1855008961529600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078667.377947:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961529600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078667.377955:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078667.377957:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078667.377961:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961529600:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078667.377989:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078664/real 1769078667] req@ffff895e44e2b480 x1855008961529728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078667.378001:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961529728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078667.378008:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078667.378011:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078667.378014:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961529728:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078667.568106:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078667.951924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078668.592050:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078668.975828:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078669.423882:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078669.423891:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078669.423894:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078669.423902:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4351 00000100:00080000:3.0:1769078669.423910:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078669.423915:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078669.423942:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961529856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078669.423964:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078669.423967:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078669.423969:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078669.423972:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4351 00000100:00080000:3.0:1769078669.423988:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078669.423992:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078669.424001:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078669.424011:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961529856:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078669.424012:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961529984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078669.424022:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078669.424024:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078669.424026:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078669.424029:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4351 00000100:00080000:3.0:1769078669.424032:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078669.424036:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078669.424040:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078669.424048:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961530112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078669.424073:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078669.424111:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078669/real 1769078669] req@ffff895e453d4380 x1855008961529856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078669.424129:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961529856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078669.424138:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078669.424142:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078669.424146:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961529856:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078669.424509:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961529984:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078669.424536:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078669.424550:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078669/real 1769078669] req@ffff895e453d5180 x1855008961529984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078669.424569:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961529984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078669.424576:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078669.424579:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078669.424582:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961529984:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078669.424589:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961530112:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078669.424607:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078669.424616:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078669/real 1769078669] req@ffff895e453d5880 x1855008961530112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078669.424627:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961530112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078669.424635:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078669.424637:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078669.424640:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961530112:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078669.616360:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078670.001000:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078670.640017:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078671.023866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078671.665553:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078672.047871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078672.689567:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078673.071849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078673.712081:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078674.095950:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078674.544355:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078674.544366:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078674.544379:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078674.544390:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4357 00000100:00080000:2.0:1769078674.544400:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078674.544407:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078674.544444:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961530240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078674.544470:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078674.544488:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078674.544490:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078674.544495:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4357 00000100:00080000:2.0:1769078674.544499:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078674.544504:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078674.544513:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078674.544521:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961530240:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078674.544528:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961530368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078674.544541:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078674.544544:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078674.544546:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078674.544550:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4357 00000100:00080000:2.0:1769078674.544554:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078674.544558:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078674.544563:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078674.544574:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961530496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078674.544667:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078674.544725:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078674/real 1769078674] req@ffff895e44e2b100 x1855008961530240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078674.544772:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961530240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078674.544860:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078674.544865:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078674.544869:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961530240:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078674.544895:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961530368:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078674.544921:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078674.544934:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078674/real 1769078674] req@ffff895e44e2a300 x1855008961530368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078674.544945:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961530368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078674.544953:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078674.544955:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078674.544958:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961530368:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078674.544963:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961530496:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078674.544982:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078674.544992:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078674/real 1769078674] req@ffff895e44e2aa00 x1855008961530496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078674.545004:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961530496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078674.545023:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078674.545027:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078674.545030:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961530496:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078674.736128:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078675.119886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078675.760318:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078676.143832:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078676.783982:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078677.167853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078677.808044:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078678.191865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078678.831933:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078679.216000:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078679.663976:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078679.663986:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078679.663991:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078679.663999:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4362 00000100:00080000:3.0:1769078679.664011:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078679.664018:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078679.664053:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961530624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078679.664155:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078679.664172:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078679.664174:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078679.664178:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4362 00000100:00080000:3.0:1769078679.664182:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078679.664186:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078679.664194:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078679.664205:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961530752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078679.664227:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078679.664230:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078679.664232:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078679.664235:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4362 00000100:00080000:3.0:1769078679.664239:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078679.664242:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078679.664246:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078679.664256:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961530880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078679.664433:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961530624:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078679.664490:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078679.664527:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078679/real 1769078679] req@ffff895e453d4700 x1855008961530624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078679.664543:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961530624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078679.664552:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078679.664556:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078679.664560:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961530624:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078679.664566:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961530752:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078679.664583:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078679.664594:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078679/real 1769078679] req@ffff895e453d7100 x1855008961530752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078679.664617:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961530752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078679.664625:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078679.664628:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078679.664631:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961530752:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078679.664637:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961530880:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078679.664655:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078679.664664:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078679/real 1769078679] req@ffff895e453d7b80 x1855008961530880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078679.664674:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961530880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078679.664681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078679.664684:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078679.664686:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961530880:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078679.855961:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078680.239854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078680.880788:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078681.265545:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078681.905693:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078682.287861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078682.927855:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078683.311863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078683.951977:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078684.335872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078684.783925:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078684.783934:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078684.783938:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078684.783947:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4367 00000100:00080000:2.0:1769078684.783957:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078684.783963:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078684.783996:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961531008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078684.784036:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078684.784039:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078684.784040:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078684.784044:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4367 00000100:00080000:2.0:1769078684.784048:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078684.784052:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078684.784061:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078684.784072:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961531136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078684.784082:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078684.784086:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078684.784088:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078684.784091:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4367 00000100:00080000:2.0:1769078684.784095:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078684.784100:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961531008:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769078684.784100:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078684.784105:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078684.784114:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961531264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1769078684.784189:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078684.784226:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078684/real 1769078684] req@ffff895e44e28000 x1855008961531008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078684.784249:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961531008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078684.784259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078684.784263:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078684.784266:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961531008:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078684.784296:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961531136:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078684.784323:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078684.784336:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078684/real 1769078684] req@ffff895e44e28380 x1855008961531136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078684.784348:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961531136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078684.784356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078684.784359:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078684.784362:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961531136:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078684.784368:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961531264:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078684.784387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078684.784396:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078684/real 1769078684] req@ffff895e44e29880 x1855008961531264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078684.784419:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961531264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078684.784426:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078684.784429:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078684.784431:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961531264:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078684.975943:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078685.359855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078686.000015:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078686.383886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078687.023940:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078687.407984:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078688.047939:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078688.431836:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078689.072084:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078689.455864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078689.903989:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078689.904000:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078689.904005:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078689.904015:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4372 00000100:00080000:3.0:1769078689.904026:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078689.904044:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078689.904081:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961531392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078689.904122:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078689.904125:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078689.904127:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078689.904131:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4372 00000100:00080000:3.0:1769078689.904135:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078689.904139:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078689.904147:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078689.904159:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961531520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078689.904167:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078689.904171:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078689.904172:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078689.904175:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4372 00000100:00080000:3.0:1769078689.904178:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078689.904182:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078689.904186:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078689.904195:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961531648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078689.904229:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961531392:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078689.905351:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078689.905390:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078689/real 1769078689] req@ffff895e453d5500 x1855008961531392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078689.905407:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961531392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078689.905418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078689.905423:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078689.905427:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961531392:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078689.905438:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961531520:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078689.905471:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078689.905494:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078689/real 1769078689] req@ffff895e453d4e00 x1855008961531520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078689.905506:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961531520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078689.905515:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078689.905518:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078689.905522:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961531520:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078689.905527:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961531648:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078689.905545:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078689.906363:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078689/real 1769078689] req@ffff895e453d4a80 x1855008961531648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078689.906378:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961531648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078689.906386:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078689.906389:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078689.906393:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961531648:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078690.097500:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078690.479866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078691.120224:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078691.506876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078692.144029:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078692.527893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078693.172169:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078693.551865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078694.191984:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078694.575872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078695.025227:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078695.025238:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078695.025242:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078695.025252:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4377 00000100:00080000:0.0:1769078695.025263:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078695.025271:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078695.025308:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961531776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078695.025384:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961531776:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078695.025461:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769078695.025518:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078695.025522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078695.027636:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078695.027645:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4377 00000100:00080000:0.0:1769078695.027653:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078695.027659:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078695.027669:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078695.027695:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961531904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078695.028126:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078695/real 1769078695] req@ffff895e4383ed80 x1855008961531776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078695.028148:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961531776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078695.028157:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078695.028161:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078695.028165:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961531776:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078695.028190:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961531904:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078695.028231:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078695.028244:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078695/real 1769078695] req@ffff895e4383df80 x1855008961531904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078695.028255:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961531904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078695.028263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078695.028266:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078695.028269:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961531904:192.168.204.117@tcp:38:kworker.0 00000100:00080000:0.0:1769078695.035232:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078695.035243:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078695.035248:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078695.035258:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4377 00000100:00080000:0.0:1769078695.035270:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078695.035276:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078695.035305:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078695.035343:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961532032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078695.036433:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961532032:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078695.036882:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078695.036914:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078695/real 1769078695] req@ffff895e4383d180 x1855008961532032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078695.036962:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961532032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078695.036973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078695.036978:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078695.036981:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961532032:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078695.220114:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078695.599905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078696.240132:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078696.626479:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078697.269362:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078697.648139:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078698.293705:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078698.671870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078699.312065:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078699.698957:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078700.144440:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078700.144449:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078700.144453:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078700.144462:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4382 00000100:00080000:0.0:1769078700.144473:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078700.144480:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078700.144512:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961532160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078700.144541:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078700.144545:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078700.144546:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078700.144550:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4382 00000100:00080000:0.0:1769078700.144553:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078700.144557:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078700.144564:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078700.144574:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961532288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078700.144593:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078700.144608:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078700.144610:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078700.144613:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4382 00000100:00080000:0.0:1769078700.144617:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078700.144620:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078700.144624:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078700.144632:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961532416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078700.144682:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961532160:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769078700.144738:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078700.144794:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078700/real 1769078700] req@ffff895e4383d180 x1855008961532160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078700.144821:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961532160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078700.144829:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078700.144833:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078700.144836:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961532160:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078700.144844:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961532288:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078700.144878:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078700.144890:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078700/real 1769078700] req@ffff895e4383df80 x1855008961532288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078700.144900:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961532288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078700.144907:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078700.144910:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078700.144913:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961532288:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078700.144918:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961532416:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078700.144937:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078700.144947:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078700/real 1769078700] req@ffff895e4383ed80 x1855008961532416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078700.144957:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961532416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078700.144973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078700.144977:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078700.144980:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961532416:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078700.335982:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078700.721905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078701.360938:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078701.743896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078702.384413:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078702.770234:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078703.408265:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078703.791871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078704.432237:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078704.816024:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078705.263885:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078705.263896:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078705.263901:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078705.263911:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4387 00000100:00080000:0.0:1769078705.263923:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078705.263930:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078705.263977:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961532544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078705.264052:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961532544:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769078705.264125:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769078705.264156:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078705.264160:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078705.264162:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078705.264166:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4387 00000100:00080000:0.0:1769078705.264170:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078705.264188:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078705.264197:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078705.264209:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961532672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078705.264409:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078705/real 1769078705] req@ffff895e4383d880 x1855008961532544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078705.264426:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961532544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078705.264435:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078705.264438:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078705.264442:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961532544:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078705.264465:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961532672:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078705.264487:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078705.264499:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078705/real 1769078705] req@ffff895e4383e680 x1855008961532672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078705.264510:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961532672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078705.264519:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078705.264522:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078705.264525:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961532672:192.168.204.117@tcp:38:kworker.0 00000100:00080000:0.0:1769078705.264631:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078705.264635:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078705.264637:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078705.264642:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4387 00000100:00080000:0.0:1769078705.264646:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078705.264649:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078705.264665:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078705.264677:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961532800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078705.264724:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961532800:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078705.264769:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078705.264930:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078705/real 1769078705] req@ffff895e4383c700 x1855008961532800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078705.264942:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961532800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078705.264949:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078705.264952:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078705.265015:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961532800:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078705.455936:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078705.840010:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078706.651580:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078706.866892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078707.503992:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078707.888714:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078708.528009:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078708.915584:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078709.559161:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078709.935884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078710.383927:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078710.383939:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078710.383961:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078710.383971:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4392 00000100:00080000:1.0:1769078710.383982:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078710.383989:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078710.384021:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961532928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078710.384055:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078710.384058:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078710.384060:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078710.384064:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4392 00000100:00080000:1.0:1769078710.384068:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078710.384072:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078710.384081:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078710.384092:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961533056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078710.384102:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078710.384104:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078710.384106:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078710.384110:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4392 00000100:00080000:1.0:1769078710.384113:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078710.384116:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078710.384121:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078710.384140:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961533184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078710.384365:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961532928:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769078710.384425:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078710.384460:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078710/real 1769078710] req@ffff895e4713fb80 x1855008961532928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078710.384485:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961532928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078710.384496:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078710.384500:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078710.384504:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961532928:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078710.384511:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961533056:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078710.384531:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078710.384542:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078710/real 1769078710] req@ffff895e4713c700 x1855008961533056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078710.384562:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961533056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078710.385306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078710.385314:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078710.385319:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961533056:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078710.385328:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961533184:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078710.385360:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078710.385374:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078710/real 1769078710] req@ffff895e4713c000 x1855008961533184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078710.385387:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961533184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078710.385395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078710.385398:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078710.385402:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961533184:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078710.577927:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078710.960304:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078711.602030:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078711.983869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078712.627914:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078713.007908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078713.649077:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078714.034069:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078714.674740:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078715.055910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078715.503934:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078715.503949:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078715.503954:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078715.503968:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4398 00000100:00080000:1.0:1769078715.503983:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078715.503994:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078715.504049:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961533312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078715.504088:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078715.504091:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078715.504093:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078715.504098:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4398 00000100:00080000:1.0:1769078715.504101:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078715.504105:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078715.504116:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078715.504136:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961533440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078715.504145:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078715.504166:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078715.504168:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078715.504172:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4398 00000100:00080000:1.0:1769078715.504176:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078715.504180:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078715.504185:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078715.504195:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961533568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078715.505818:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961533312:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769078715.505926:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078715.505979:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078715/real 1769078715] req@ffff895e4713ed80 x1855008961533312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078715.506001:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961533312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078715.506044:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078715.506049:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078715.506054:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961533312:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078715.506062:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961533440:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078715.506085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078715.506096:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078715/real 1769078715] req@ffff895e4713d880 x1855008961533440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078715.506106:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961533440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078715.506113:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078715.506117:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078715.506120:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961533440:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078715.506135:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961533568:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078715.506152:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078715.506160:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078715/real 1769078715] req@ffff895e4713e300 x1855008961533568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078715.506169:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961533568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078715.506176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078715.506179:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078715.506182:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961533568:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078715.696944:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078716.079964:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078716.719952:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078717.103878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078717.744439:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078718.127856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078718.768589:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078719.156783:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078719.794737:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078720.177266:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078720.624001:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078720.624013:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078720.624017:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078720.624038:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4403 00000100:00080000:1.0:1769078720.624050:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078720.624057:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078720.624096:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961533696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078720.624128:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078720.624132:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078720.624134:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078720.624152:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4403 00000100:00080000:1.0:1769078720.624156:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078720.624161:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078720.624170:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078720.624182:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961533824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078720.624192:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078720.624196:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078720.624197:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078720.624201:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4403 00000100:00080000:1.0:1769078720.624204:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078720.624208:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078720.624212:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078720.624223:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961533952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078720.624393:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961533696:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769078720.624454:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078720.624489:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078720/real 1769078720] req@ffff895e4713f100 x1855008961533696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078720.624508:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961533696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078720.624528:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078720.624533:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078720.624537:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961533696:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769078720.624544:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961533824:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078720.624567:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078720.624579:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078720/real 1769078720] req@ffff895e4713ea00 x1855008961533824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078720.624590:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961533824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078720.624599:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078720.624603:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078720.624606:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961533824:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769078720.624612:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961533952:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769078720.624630:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769078720.624640:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078720/real 1769078720] req@ffff895e4713ce00 x1855008961533952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078720.624672:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961533952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078720.624680:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078720.624683:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078720.624686:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961533952:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078720.816386:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078721.204616:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078721.842463:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078722.224864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078722.864177:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078723.247872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078723.887922:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078724.271874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078724.912970:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078725.295878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078725.744104:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078725.744119:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078725.744125:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078725.744167:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4408 00000100:00080000:1.0:1769078725.744180:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078725.744190:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078725.744237:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961534080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078725.744335:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961534080:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769078725.744419:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:1.0:1769078725.744640:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078725.744645:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078725.744647:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078725.744651:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4408 00000100:00080000:1.0:1769078725.744655:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078725.744660:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078725.744671:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078725.744684:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961534208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769078725.748774:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078725/real 1769078725] req@ffff895e4713d500 x1855008961534080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769078725.748807:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961534080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769078725.748820:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769078725.748825:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769078725.748830:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961534080:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078725.751354:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961534208:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078725.751423:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078725.751461:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078725/real 1769078725] req@ffff895e4713f480 x1855008961534208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078725.751501:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961534208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078725.751513:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078725.751518:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078725.751521:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961534208:192.168.204.117@tcp:38:kworker.0 00000100:00080000:1.0:1769078725.752221:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078725.752227:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078725.752230:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078725.752237:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4408 00000100:00080000:1.0:1769078725.752244:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078725.752250:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078725.752260:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078725.752288:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961534336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078725.752428:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961534336:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078725.752473:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078725.752488:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078725/real 1769078725] req@ffff895e4713ca80 x1855008961534336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078725.752500:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961534336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078725.752508:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078725.752511:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078725.752514:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961534336:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078725.941693:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078726.319928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078726.964592:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078727.344649:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078727.984433:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078728.367839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078729.008605:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078729.392728:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078730.032743:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078730.415870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078730.863871:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078730.863884:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078730.863888:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078730.863898:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4413 00000100:00080000:1.0:1769078730.863909:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078730.863916:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078730.863964:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961534464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078730.863996:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078730.864000:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078730.864002:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078730.864006:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4413 00000100:00080000:1.0:1769078730.864009:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078730.864013:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078730.864021:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078730.864031:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961534592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078730.864040:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078730.864056:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078730.864058:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078730.864061:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4413 00000100:00080000:1.0:1769078730.864065:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078730.864069:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078730.864074:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078730.864084:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961534720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078730.864186:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961534464:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078730.864277:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961534592:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078730.864301:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961534720:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078731.055996:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078731.440482:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078732.080769:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078732.463916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078733.104300:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078733.489863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769078733.936108:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769078733.936149:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078733.936167:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078733.936207:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078733.936216:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769078733.936441:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078730/real 1769078733] req@ffff895e4713f800 x1855008961534464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078733.936496:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961534464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078733.936508:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078733.936514:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078733.936517:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961534464:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078733.936525:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078730/real 1769078733] req@ffff895e4713dc00 x1855008961534592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078733.936536:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961534592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078733.936550:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078733.936553:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078733.936557:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961534592:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078733.936563:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078730/real 1769078733] req@ffff895e4713df80 x1855008961534720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078733.936574:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961534720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078733.936581:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078733.936584:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078733.936587:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961534720:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078734.129945:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078734.514140:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078735.152042:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078735.535916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078735.985727:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078735.985740:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078735.985775:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078735.985787:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4418 00000100:00080000:1.0:1769078735.985799:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078735.985808:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078735.985849:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961534848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078735.985876:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078735.985879:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078735.985881:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078735.985902:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4418 00000100:00080000:1.0:1769078735.985906:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078735.985910:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078735.985920:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078735.985932:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961534976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078735.985942:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078735.985944:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078735.985946:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078735.985950:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4418 00000100:00080000:1.0:1769078735.985965:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078735.985969:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078735.985973:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078735.985984:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961535104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078735.986210:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961534848:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078735.986284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078735.986319:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078735/real 1769078735] req@ffff895e4713c380 x1855008961534848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078735.986338:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961534848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078735.986348:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078735.986351:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078735.986355:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961534848:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078735.986363:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961534976:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078735.986393:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078735.986404:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078735/real 1769078735] req@ffff895e4713d180 x1855008961534976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078735.986416:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961534976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078735.986423:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078735.986426:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078735.986429:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961534976:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078735.986434:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961535104:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078735.986450:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078735.986459:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078735/real 1769078735] req@ffff895e4713df80 x1855008961535104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078735.986479:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961535104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078735.986487:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078735.986490:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078735.986493:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961535104:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078736.175968:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078736.559912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078737.203379:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078737.583862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078738.224104:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078738.607880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078739.247925:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078739.631860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078740.272533:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078740.655894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078741.103905:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078741.103916:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078741.103938:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078741.103948:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4423 00000100:00080000:2.0:1769078741.103960:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078741.103968:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078741.104004:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961535232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078741.104225:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961535232:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078741.104285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:2.0:1769078741.104318:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078741.104321:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078741.104323:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078741.104328:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4423 00000100:00080000:2.0:1769078741.104332:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078741.104336:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078741.104346:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078741.104360:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961535360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078741.108073:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078741/real 1769078741] req@ffff895e44e29880 x1855008961535232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078741.108101:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961535232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078741.108109:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078741.108113:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078741.108116:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961535232:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078741.108148:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961535360:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078741.108216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078741.108233:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078741/real 1769078741] req@ffff895e44e28380 x1855008961535360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078741.108246:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961535360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078741.108270:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078741.108273:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078741.108275:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961535360:192.168.204.117@tcp:38:kworker.0 00000100:00080000:2.0:1769078741.108376:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078741.108381:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078741.108384:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078741.108389:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4423 00000100:00080000:2.0:1769078741.108394:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078741.108400:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078741.108411:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078741.108434:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961535488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078741.108508:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961535488:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078741.108526:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078741.108534:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078741/real 1769078741] req@ffff895e44e28000 x1855008961535488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078741.108541:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961535488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078741.108546:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078741.108548:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078741.108550:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961535488:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078741.296353:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078741.681345:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078742.320030:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078742.708339:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078743.344843:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078743.727871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078744.368180:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078744.752003:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078745.392152:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078745.775967:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078746.226867:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078746.226878:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078746.226882:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078746.226893:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4428 00000100:00080000:1.0:1769078746.226904:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078746.226911:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078746.226946:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961535616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078746.226977:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078746.226980:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078746.226982:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078746.226986:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4428 00000100:00080000:1.0:1769078746.226989:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078746.226993:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078746.227002:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078746.227013:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961535744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078746.227022:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078746.227025:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078746.227027:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078746.227030:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4428 00000100:00080000:1.0:1769078746.227033:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078746.227050:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078746.227054:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078746.227064:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961535872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078746.227160:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961535616:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078746.227218:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078746.227254:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078746/real 1769078746] req@ffff895e4713dc00 x1855008961535616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078746.227270:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961535616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078746.227279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078746.227283:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078746.227293:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961535616:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078746.227301:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961535744:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078746.227325:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078746.227338:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078746/real 1769078746] req@ffff895e4713f800 x1855008961535744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078746.227350:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961535744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078746.227358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078746.227361:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078746.227364:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961535744:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078746.227370:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961535872:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078746.227388:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078746.227398:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078746/real 1769078746] req@ffff895e4713ca80 x1855008961535872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078746.227407:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961535872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078746.227414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078746.227417:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078746.227420:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961535872:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078746.416066:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078746.801617:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078747.440687:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078747.825229:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078748.464099:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078748.848410:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078749.487952:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078749.871879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078750.513477:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078750.897250:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078751.344302:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078751.344313:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078751.344318:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078751.344327:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4433 00000100:00080000:1.0:1769078751.344338:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078751.344345:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078751.344381:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961536000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078751.344412:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078751.344415:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078751.344417:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078751.344421:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4433 00000100:00080000:1.0:1769078751.344424:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078751.344428:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078751.344437:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078751.344465:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961536128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078751.344474:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078751.344477:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078751.344479:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078751.344482:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4433 00000100:00080000:1.0:1769078751.344486:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078751.344489:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078751.344493:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078751.344501:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961536256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078751.345828:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961536000:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078751.345928:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078751.347154:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078751/real 1769078751] req@ffff895e4713f480 x1855008961536000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078751.347193:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961536000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078751.347206:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078751.347212:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078751.347217:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961536000:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078751.347232:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961536128:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078751.347285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078751.347310:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078751/real 1769078751] req@ffff895e4713d500 x1855008961536128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078751.347322:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961536128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078751.347330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078751.347334:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078751.347337:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961536128:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078751.347343:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961536256:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078751.347363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078751.347374:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078751/real 1769078751] req@ffff895e4713ce00 x1855008961536256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078751.347386:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961536256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078751.348821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078751.348831:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078751.348838:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961536256:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078751.541884:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078751.920221:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078752.567630:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078752.943913:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078753.584010:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078753.970143:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078754.608105:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078754.991981:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078755.632081:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078756.017493:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078756.463986:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078756.463997:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078756.464001:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078756.464010:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4439 00000100:00080000:3.0:1769078756.464022:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078756.464029:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078756.464067:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961536384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078756.464106:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078756.464110:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078756.464111:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078756.464115:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4439 00000100:00080000:3.0:1769078756.464118:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078756.464122:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078756.464131:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078756.464142:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961536512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078756.464151:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078756.464154:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078756.464170:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078756.464174:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4439 00000100:00080000:3.0:1769078756.464178:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078756.464182:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078756.464187:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078756.464198:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961536640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078756.464304:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961536384:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078756.464366:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078756.464404:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078756/real 1769078756] req@ffff895e453d7480 x1855008961536384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078756.464421:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961536384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078756.464430:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078756.464440:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078756.464444:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961536384:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078756.464451:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961536512:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078756.464474:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078756.464487:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078756/real 1769078756] req@ffff895e453d5f80 x1855008961536512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078756.464516:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961536512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078756.464524:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078756.464527:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078756.464531:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961536512:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078756.464537:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961536640:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078756.464555:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078756.464565:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078756/real 1769078756] req@ffff895e453d5c00 x1855008961536640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078756.464574:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961536640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078756.464593:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078756.464596:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078756.464599:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961536640:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078756.656048:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078757.039874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078757.681976:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078758.063839:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078758.706602:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078759.087852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078759.728091:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078760.112399:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078760.751990:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078761.136792:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078761.584043:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078761.584055:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078761.584060:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078761.584071:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4444 00000100:00080000:3.0:1769078761.584084:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078761.584091:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078761.584131:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961536768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078761.584168:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078761.584171:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078761.584173:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078761.584177:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4444 00000100:00080000:3.0:1769078761.584181:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078761.584185:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078761.584194:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078761.584224:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961536896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078761.584233:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078761.584236:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078761.584238:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078761.584241:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4444 00000100:00080000:3.0:1769078761.584245:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078761.584248:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078761.584252:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078761.584260:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961537024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078761.584706:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961536768:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078761.584794:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078761.584842:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078761/real 1769078761] req@ffff895e453d7800 x1855008961536768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078761.584861:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961536768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078761.584871:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078761.584874:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078761.584878:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961536768:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078761.584885:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961536896:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078761.584906:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078761.584919:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078761/real 1769078761] req@ffff895e453d5c00 x1855008961536896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078761.584930:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961536896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078761.584938:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078761.584941:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078761.584944:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961536896:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078761.584949:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961537024:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078761.584992:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078761.585002:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078761/real 1769078761] req@ffff895e453d5f80 x1855008961537024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078761.585012:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961537024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078761.585019:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078761.585022:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078761.585024:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961537024:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078761.776671:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078762.159863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078762.801092:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078763.183929:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078763.824001:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078764.209203:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078764.848130:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078765.231906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078765.872729:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078766.258119:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078766.704043:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078766.704056:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078766.704060:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078766.704070:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4449 00000100:00080000:2.0:1769078766.704081:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078766.704088:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078766.704137:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961537152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078766.704168:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078766.704172:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078766.704173:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078766.704177:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4449 00000100:00080000:2.0:1769078766.704181:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078766.704185:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078766.704194:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078766.704206:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961537280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078766.704215:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078766.704218:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078766.704220:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078766.704224:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4449 00000100:00080000:2.0:1769078766.704228:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078766.704231:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078766.704252:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078766.704263:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961537408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078766.704353:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961537152:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078766.704411:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078766.704442:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078766/real 1769078766] req@ffff895e44e2aa00 x1855008961537152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078766.704454:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961537152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078766.704461:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078766.704463:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078766.704465:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961537152:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078766.704471:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961537280:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078766.704490:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078766.704500:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078766/real 1769078766] req@ffff895e44e2a300 x1855008961537280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078766.704526:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961537280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078766.704534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078766.704537:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078766.704541:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961537280:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078766.704546:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961537408:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078766.704563:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078766.704570:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078766/real 1769078766] req@ffff895e44e2b100 x1855008961537408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078766.704577:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961537408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078766.704582:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078766.704584:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078766.704585:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961537408:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078766.896770:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078767.279912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078767.921170:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078768.303933:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078768.946089:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078769.327874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078769.968126:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078770.351878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078770.996707:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078771.375868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078771.824141:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078771.824152:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078771.824156:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078771.824166:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4454 00000100:00080000:2.0:1769078771.824177:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078771.824185:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078771.824224:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961537536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078771.824258:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078771.824261:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078771.824263:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078771.824267:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4454 00000100:00080000:2.0:1769078771.824271:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078771.824275:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078771.824284:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078771.824296:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961537664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078771.824305:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078771.824307:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078771.824309:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078771.824312:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4454 00000100:00080000:2.0:1769078771.824333:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078771.824337:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078771.824341:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078771.824351:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961537792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078771.824903:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961537536:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078771.824967:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078771.825478:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078771/real 1769078771] req@ffff895e44e2b480 x1855008961537536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078771.825496:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961537536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078771.825516:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078771.825520:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078771.825524:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961537536:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078771.825533:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961537664:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078771.825560:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078771.825572:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078771/real 1769078771] req@ffff895e44e28e00 x1855008961537664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078771.825583:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961537664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078771.825591:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078771.825595:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078771.825598:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961537664:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078771.826084:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961537792:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078771.826110:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078771.826122:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078771/real 1769078771] req@ffff895e44e29f80 x1855008961537792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078771.826135:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961537792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078771.826143:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078771.826147:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078771.826150:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961537792:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078772.021255:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078772.399847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078773.044119:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078773.423880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078774.063989:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078774.447877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078775.088631:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078775.472067:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078776.111984:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078776.496837:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078776.943867:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078776.943878:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078776.943883:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078776.943893:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4459 00000100:00080000:2.0:1769078776.943905:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078776.943912:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078776.943948:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961537920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078776.943988:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078776.943991:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078776.943993:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078776.943997:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4459 00000100:00080000:2.0:1769078776.944000:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078776.944004:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078776.944012:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078776.944022:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961538048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078776.944032:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078776.944051:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078776.944053:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078776.944056:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4459 00000100:00080000:2.0:1769078776.944060:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078776.944064:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078776.944068:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078776.944076:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961538176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078776.944165:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961537920:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078776.944224:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078776.944265:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078776/real 1769078776] req@ffff895e44e29500 x1855008961537920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078776.944282:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961537920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078776.944291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078776.944295:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078776.944299:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961537920:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078776.944306:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961538048:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078776.944323:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078776.944334:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078776/real 1769078776] req@ffff895e44e28a80 x1855008961538048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078776.944345:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961538048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078776.944352:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078776.944355:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078776.944358:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961538048:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078776.944363:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961538176:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078776.944390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078776.944400:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078776/real 1769078776] req@ffff895e44e29180 x1855008961538176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078776.944411:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961538176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078776.944419:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078776.944422:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078776.944425:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961538176:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078777.135924:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078777.521309:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078778.160218:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078778.544626:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078779.184665:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078779.567923:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078780.208213:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078780.591955:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078781.232028:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078781.615878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078782.068170:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078782.068182:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078782.068186:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078782.068197:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4464 00000100:00080000:3.0:1769078782.068208:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078782.068215:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078782.068252:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961538304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078782.068285:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078782.068289:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078782.068291:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078782.068295:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4464 00000100:00080000:3.0:1769078782.068299:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078782.068303:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078782.068312:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078782.068324:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961538432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078782.068334:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078782.068336:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078782.068338:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078782.068341:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4464 00000100:00080000:3.0:1769078782.068344:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769078782.068499:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961538304:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078782.068562:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078782.068683:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078782/real 1769078782] req@ffff895e453d7480 x1855008961538304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078782.068705:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961538304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078782.068715:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078782.068718:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078782.068722:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961538304:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078782.068729:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961538432:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078782.068932:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078782.068947:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078782/real 1769078782] req@ffff895e453d4a80 x1855008961538432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078782.069053:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961538432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078782.069062:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078782.069066:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078782.069069:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961538432:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769078782.069159:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078782.069167:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078782.069180:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961538560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078782.071262:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961538560:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078782.071328:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078782.071355:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078782/real 1769078782] req@ffff895e453d4e00 x1855008961538560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078782.073201:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961538560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078782.073215:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078782.073220:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078782.073224:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961538560:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078782.256235:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078782.639875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078783.280513:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078783.666852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078784.304601:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078784.687862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078785.328093:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078785.711852:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078786.352271:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078786.736022:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078787.184057:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078787.184067:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078787.184072:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078787.184081:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4469 00000100:00080000:2.0:1769078787.184093:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078787.184100:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078787.184148:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961538688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078787.184187:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078787.184191:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078787.184193:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078787.184197:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4469 00000100:00080000:2.0:1769078787.184201:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078787.184205:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078787.184214:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078787.184225:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961538816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078787.184249:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078787.184252:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078787.184254:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078787.184258:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4469 00000100:00080000:2.0:1769078787.184262:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078787.184265:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078787.184270:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078787.184418:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961538944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078787.184513:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961538688:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078787.184578:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078787.184627:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078787/real 1769078787] req@ffff895e44e2bb80 x1855008961538688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078787.184645:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961538688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078787.184661:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078787.184980:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078787.184985:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961538688:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078787.184994:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961538816:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078787.185021:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078787.185035:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078787/real 1769078787] req@ffff895e44e2b800 x1855008961538816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078787.185048:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961538816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078787.185057:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078787.185061:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078787.185064:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961538816:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078787.185070:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961538944:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078787.185088:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078787.185099:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078787/real 1769078787] req@ffff895e44e29c00 x1855008961538944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078787.185110:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961538944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078787.185118:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078787.185122:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078787.185125:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961538944:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078787.376086:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078787.760069:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078788.400189:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078788.785634:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078789.424050:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078789.807936:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078790.450809:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078790.836167:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078791.472102:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078791.855869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078792.303869:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078792.303879:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078792.303883:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078792.303893:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4474 00000100:00080000:2.0:1769078792.303904:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078792.303911:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078792.303948:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961539072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078792.303980:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078792.303984:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078792.303985:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078792.303990:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4474 00000100:00080000:2.0:1769078792.303993:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078792.303997:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078792.304018:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078792.304030:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961539200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078792.304040:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078792.304043:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078792.304045:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078792.304049:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4474 00000100:00080000:2.0:1769078792.304052:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078792.304056:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078792.304061:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078792.304070:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961539328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078792.304162:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961539072:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078792.304221:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078792.304266:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078792/real 1769078792] req@ffff895e44e29c00 x1855008961539072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078792.304284:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961539072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078792.304295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078792.304299:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078792.304302:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961539072:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078792.304320:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961539200:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078792.304341:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078792.304353:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078792/real 1769078792] req@ffff895e44e2b800 x1855008961539200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078792.304364:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961539200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078792.304371:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078792.304374:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078792.304377:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961539200:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078792.304383:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961539328:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078792.304410:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078792.304421:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078792/real 1769078792] req@ffff895e44e2bb80 x1855008961539328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078792.304432:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961539328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078792.304438:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078792.304441:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078792.304444:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961539328:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078792.497024:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078792.879921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078793.519970:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078793.903865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078794.547285:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078794.927882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078795.572334:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078795.951871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078796.595090:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078796.975859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078797.426062:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078797.426072:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078797.426077:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078797.426087:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4479 00000100:00080000:3.0:1769078797.426097:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078797.426104:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078797.426139:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961539456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078797.426173:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078797.426189:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078797.426191:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078797.426196:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4479 00000100:00080000:3.0:1769078797.426200:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078797.426204:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078797.426213:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078797.426236:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961539584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078797.426246:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078797.426249:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078797.426250:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078797.426253:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4479 00000100:00080000:3.0:1769078797.426257:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078797.426260:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078797.426265:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078797.426274:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961539712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078797.426709:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961539456:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078797.426831:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961539584:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078797.426857:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961539712:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078797.616041:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078797.999843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078798.641965:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078799.023902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078799.663968:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078800.047883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769078800.496277:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769078800.496321:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078800.496339:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078800.496535:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078800.496545:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:1.0:1769078800.500150:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078797/real 1769078800] req@ffff895e453d5500 x1855008961539456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078800.500197:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961539456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078800.500209:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078800.500215:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078800.500219:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961539456:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078800.500228:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078797/real 1769078800] req@ffff895e453d7b80 x1855008961539584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078800.500264:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961539584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078800.500273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078800.500277:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078800.500280:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961539584:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078800.500824:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078797/real 1769078800] req@ffff895e453d7100 x1855008961539712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078800.500840:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961539712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078800.500848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078800.500851:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078800.500855:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961539712:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078800.690341:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078801.071865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078801.712004:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078802.096192:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078802.543993:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078802.544004:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078802.544008:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078802.544019:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4485 00000100:00080000:2.0:1769078802.544030:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078802.544037:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078802.544086:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961539840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078802.544134:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078802.544147:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078802.544149:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078802.544153:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4485 00000100:00080000:2.0:1769078802.544157:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078802.544162:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078802.544171:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078802.544184:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961539968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078802.544194:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078802.544197:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078802.544198:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078802.544201:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4485 00000100:00080000:2.0:1769078802.544204:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078802.544208:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078802.544211:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078802.544220:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961540096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078802.544694:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961539840:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078802.544785:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078802.544823:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078802/real 1769078802] req@ffff895e44e29180 x1855008961539840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078802.544839:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961539840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078802.544849:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078802.544853:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078802.544865:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961539840:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078802.544873:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961539968:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078802.544897:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078802.544909:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078802/real 1769078802] req@ffff895e44e28a80 x1855008961539968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078802.544920:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961539968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078802.544927:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078802.544930:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078802.544933:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961539968:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078802.544939:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961540096:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078802.544954:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078802.544962:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078802/real 1769078802] req@ffff895e44e29500 x1855008961540096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078802.544972:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961540096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078802.544979:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078802.544982:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078802.544985:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961540096:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078802.737391:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078803.119945:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078803.760214:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078804.143898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078804.785624:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078805.167871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078805.808938:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078806.192740:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078806.831974:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078807.219272:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078807.663915:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078807.663925:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078807.663935:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078807.663945:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4490 00000100:00080000:3.0:1769078807.663956:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078807.663963:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078807.663997:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961540224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078807.664030:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078807.664033:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078807.664035:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078807.664039:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4490 00000100:00080000:3.0:1769078807.664043:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078807.664046:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078807.664064:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078807.664076:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961540352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078807.664084:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078807.664087:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078807.664088:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078807.664092:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4490 00000100:00080000:3.0:1769078807.664095:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078807.664097:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078807.664116:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078807.664127:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961540480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769078807.664272:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961540224:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769078807.664334:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078807.664372:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078807/real 1769078807] req@ffff895e453d4700 x1855008961540224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078807.664388:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961540224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078807.664396:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078807.664400:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078807.664403:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961540224:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769078807.664422:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961540352:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078807.664444:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078807.664456:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078807/real 1769078807] req@ffff895e453d5880 x1855008961540352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078807.664467:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961540352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078807.664474:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078807.664477:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078807.664479:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961540352:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769078807.664485:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961540480:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769078807.664502:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769078807.664513:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078807/real 1769078807] req@ffff895e453d5180 x1855008961540480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769078807.664539:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961540480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769078807.664546:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769078807.664549:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769078807.664552:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961540480:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078807.855922:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078808.239880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078808.880558:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078809.263887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078809.903964:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078810.287845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078810.927956:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078811.311843:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078811.953390:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078812.335889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078812.783988:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078812.783999:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078812.784004:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078812.784016:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4495 00000100:00080000:1.0:1769078812.784028:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078812.784035:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078812.784078:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961540608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078812.784112:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078812.784115:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078812.784117:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078812.784121:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4495 00000100:00080000:1.0:1769078812.784125:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078812.784128:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078812.784138:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078812.784149:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961540736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078812.784174:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078812.784177:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078812.784179:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078812.784182:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4495 00000100:00080000:1.0:1769078812.784187:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078812.784191:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078812.784196:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078812.784207:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961540864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078812.784289:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961540608:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078812.784361:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078812.784414:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078812/real 1769078812] req@ffff895e4713ea00 x1855008961540608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078812.784431:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961540608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078812.784439:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078812.784443:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078812.784446:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961540608:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078812.784452:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961540736:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078812.784472:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078812.784483:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078812/real 1769078812] req@ffff895e4713f100 x1855008961540736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078812.784905:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961540736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078812.784927:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078812.784931:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078812.784935:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961540736:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078812.784942:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961540864:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078812.784980:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078812.784993:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078812/real 1769078812] req@ffff895e4713e300 x1855008961540864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078812.785004:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961540864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078812.785012:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078812.785015:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078812.785018:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961540864:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078812.976080:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078813.359879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078813.999927:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078814.383866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078815.025071:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078815.408674:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078816.047966:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078816.431977:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078817.072009:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078817.456315:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078817.904192:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078817.904204:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078817.904208:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078817.904218:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4500 00000100:00080000:1.0:1769078817.904230:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078817.904237:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078817.904275:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961540992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078817.904311:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078817.904314:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078817.904316:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078817.904320:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4500 00000100:00080000:1.0:1769078817.904340:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078817.904345:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078817.904353:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078817.904368:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961541120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078817.904378:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078817.904381:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078817.904383:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078817.904386:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4500 00000100:00080000:1.0:1769078817.904390:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078817.904394:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078817.904398:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078817.904408:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961541248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078817.904478:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961540992:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078817.904542:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078817.904594:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078817/real 1769078817] req@ffff895e4713d880 x1855008961540992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078817.904619:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961540992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078817.904629:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078817.904633:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078817.904636:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961540992:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078817.904644:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961541120:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078817.904668:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078817.904681:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078817/real 1769078817] req@ffff895e4713ed80 x1855008961541120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078817.904693:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961541120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078817.904701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078817.904704:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078817.904707:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961541120:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078817.904713:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961541248:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078817.904729:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078817.904738:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078817/real 1769078817] req@ffff895e4713c000 x1855008961541248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078817.904772:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961541248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078817.904791:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078817.904794:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078817.904797:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961541248:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078818.096615:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078818.479898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078819.119971:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078819.504210:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078820.143897:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078820.527903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078821.168666:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078821.551921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078822.192008:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078822.575896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078823.025781:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078823.025801:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078823.025806:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078823.025816:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4505 00000100:00080000:0.0:1769078823.025827:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078823.025834:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078823.025869:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961541376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078823.025939:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961541376:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078823.026000:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769078823.026025:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078823.026028:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078823.026029:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078823.026032:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4505 00000100:00080000:0.0:1769078823.026035:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078823.026038:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078823.026045:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078823.026054:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961541504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078823.034439:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078823/real 1769078823] req@ffff895e4383f100 x1855008961541376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078823.034470:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961541376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078823.034482:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078823.034486:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078823.034491:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961541376:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078823.034526:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961541504:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078823.034585:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078823.034616:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078823/real 1769078823] req@ffff895e4383c380 x1855008961541504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078823.034630:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961541504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078823.034638:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078823.034642:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078823.034645:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961541504:192.168.204.117@tcp:38:kworker.0 00000100:00080000:0.0:1769078823.035232:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078823.035237:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078823.035241:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078823.035246:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4505 00000100:00080000:0.0:1769078823.035254:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078823.035261:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078823.035272:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078823.035299:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961541632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078823.035460:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961541632:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078823.035495:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078823.035510:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078823/real 1769078823] req@ffff895e4383fb80 x1855008961541632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078823.035560:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961541632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078823.035568:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078823.035572:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078823.035575:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961541632:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078823.216044:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078823.600092:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078824.240392:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078824.623922:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078825.265930:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078825.647870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078826.287992:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078826.671864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078827.318028:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078827.696109:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078828.143941:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078828.143950:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078828.143954:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078828.143962:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4510 00000100:00080000:3.0:1769078828.143971:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078828.143976:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078828.144009:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961541760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078828.144034:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078828.144037:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078828.144038:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078828.144041:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4510 00000100:00080000:3.0:1769078828.144043:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078828.144057:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078828.144064:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078828.144072:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961541888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078828.144078:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078828.144080:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078828.144081:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078828.144083:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4510 00000100:00080000:3.0:1769078828.144085:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078828.144088:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078828.144090:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078828.144095:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961542016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078828.144212:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961541760:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078828.144273:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078828.144318:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078828/real 1769078828] req@ffff895e453d4380 x1855008961541760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078828.144335:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961541760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078828.144344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078828.144348:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078828.144352:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961541760:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078828.144359:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961541888:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078828.144387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078828.144399:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078828/real 1769078828] req@ffff895e453d6680 x1855008961541888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078828.144408:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961541888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078828.144413:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078828.144415:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078828.144417:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961541888:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078828.144420:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961542016:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078828.144433:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078828.144441:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078828/real 1769078828] req@ffff895e453d6300 x1855008961542016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078828.144451:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961542016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078828.144457:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078828.144460:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078828.144463:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961542016:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078828.338354:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078828.720474:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078829.364268:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078829.743867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078830.383935:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078830.767848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078831.408524:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078831.792230:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078832.432055:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078832.817911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078833.263916:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078833.263926:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078833.263931:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078833.263940:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4515 00000100:00080000:1.0:1769078833.263951:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078833.263958:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078833.264019:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961542144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078833.264051:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078833.264055:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078833.264056:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078833.264060:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4515 00000100:00080000:1.0:1769078833.264066:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078833.264070:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078833.264079:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078833.264099:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961542272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078833.264110:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078833.264113:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078833.264115:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078833.264118:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4515 00000100:00080000:1.0:1769078833.264122:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078833.264125:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078833.264130:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078833.264140:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961542400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078833.264739:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961542144:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078833.264826:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078833.264942:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078833/real 1769078833] req@ffff895e4713c700 x1855008961542144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078833.264963:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961542144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078833.264973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078833.264977:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078833.264981:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961542144:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078833.264988:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961542272:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078833.265022:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078833.265036:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078833/real 1769078833] req@ffff895e4713fb80 x1855008961542272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078833.265047:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961542272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078833.265056:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078833.265059:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078833.265063:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961542272:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078833.265069:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961542400:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078833.265087:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078833.265097:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078833/real 1769078833] req@ffff895e4713c000 x1855008961542400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078833.265109:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961542400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078833.265129:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078833.265133:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078833.265137:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961542400:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078833.456518:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078833.839920:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078834.480109:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078834.863944:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078835.503971:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078835.887845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078836.528213:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078836.911875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078837.552011:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078837.935874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078838.383894:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078838.383928:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078838.383932:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078838.383942:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4520 00000100:00080000:1.0:1769078838.383954:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078838.383961:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078838.383995:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961542528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078838.384026:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078838.384030:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078838.384032:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078838.384036:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4520 00000100:00080000:1.0:1769078838.384039:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078838.384043:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078838.384051:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078838.384153:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961542656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078838.384163:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078838.384166:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078838.384168:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078838.384171:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4520 00000100:00080000:1.0:1769078838.384176:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078838.384179:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078838.384184:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078838.384193:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961542784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078838.384550:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961542528:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078838.384910:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078838.384964:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078838/real 1769078838] req@ffff895e4713ed80 x1855008961542528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078838.384982:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961542528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078838.384991:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078838.384995:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078838.384999:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961542528:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078838.385007:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961542656:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078838.385029:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078838.385041:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078838/real 1769078838] req@ffff895e4713d880 x1855008961542656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078838.385051:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961542656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078838.386786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078838.386795:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078838.386802:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961542656:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078838.386816:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961542784:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078838.386892:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078838.386918:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078838/real 1769078838] req@ffff895e4713e300 x1855008961542784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078838.386933:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961542784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078838.386941:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078838.386945:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078838.386948:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961542784:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078838.575986:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078838.959899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078839.600136:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078839.986899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078840.624292:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078841.007881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078841.648083:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078842.032514:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078842.672728:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078843.056385:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078843.507488:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078843.507498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078843.507502:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078843.507512:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4526 00000100:00080000:2.0:1769078843.507538:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078843.507545:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078843.507575:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961542912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078843.507610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078843.507612:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078843.507614:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078843.507617:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4526 00000100:00080000:2.0:1769078843.507620:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078843.507624:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078843.507635:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078843.507645:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961543040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078843.507653:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078843.507656:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078843.507658:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078843.507661:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4526 00000100:00080000:2.0:1769078843.507664:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078843.507666:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078843.507670:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078843.507677:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961543168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078843.507784:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961542912:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078843.507852:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078843.507887:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078843/real 1769078843] req@ffff895e44e29f80 x1855008961542912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078843.507904:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961542912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078843.507924:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078843.507928:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078843.507932:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961542912:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078843.507939:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961543040:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078843.507960:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078843.507987:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078843/real 1769078843] req@ffff895e44e28e00 x1855008961543040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078843.507998:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961543040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078843.508005:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078843.508008:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078843.508011:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961543040:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078843.508017:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961543168:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078843.508035:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078843.508045:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078843/real 1769078843] req@ffff895e44e2b480 x1855008961543168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078843.508056:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961543168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078843.508064:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078843.508067:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078843.508070:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961543168:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078843.699111:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078844.079874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078844.719992:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078845.103883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078845.743980:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078846.127870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078846.768231:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078847.151902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078847.791949:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078848.176185:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078848.632038:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078848.632077:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078848.632081:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078848.632090:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4531 00000100:00080000:2.0:1769078848.632101:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078848.632108:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078848.632142:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961543296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078848.632172:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078848.632176:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078848.632178:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078848.632182:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4531 00000100:00080000:2.0:1769078848.632186:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078848.632190:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078848.632198:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078848.632209:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961543424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078848.632217:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078848.632221:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078848.632222:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078848.632226:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4531 00000100:00080000:2.0:1769078848.632229:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078848.632232:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078848.632236:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078848.632245:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961543552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078848.633931:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961543296:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078848.634002:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078848.634058:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078848/real 1769078848] req@ffff895e44e2b100 x1855008961543296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078848.634076:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961543296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078848.634086:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078848.634090:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078848.634093:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961543296:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078848.634100:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961543424:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078848.634121:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078848.634412:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078848/real 1769078848] req@ffff895e44e2a300 x1855008961543424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078848.634429:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961543424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078848.634437:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078848.634441:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078848.634444:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961543424:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078848.634451:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961543552:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078848.634498:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078848.634509:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078848/real 1769078848] req@ffff895e44e2aa00 x1855008961543552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078848.634520:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961543552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078848.634527:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078848.634530:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078848.634533:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961543552:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078848.816196:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078849.202574:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078849.841321:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078850.223856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078850.864002:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078851.248056:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078851.887955:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078852.271875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078852.912492:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078853.295835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078853.744170:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078853.744180:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078853.744184:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078853.744194:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4536 00000100:00080000:3.0:1769078853.744203:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078853.744211:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078853.744247:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961543680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078853.744493:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961543680:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078853.744567:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769078853.744597:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078853.744601:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078853.744603:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078853.744608:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4536 00000100:00080000:3.0:1769078853.744612:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078853.744616:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078853.744624:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078853.744649:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961543808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078853.749488:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078853/real 1769078853] req@ffff895e453d6d80 x1855008961543680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078853.749516:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961543680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078853.749526:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078853.749531:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078853.749535:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961543680:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078853.749775:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961543808:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078853.749834:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078853.749865:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078853/real 1769078853] req@ffff895e453d6300 x1855008961543808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078853.749878:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961543808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078853.749886:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078853.749889:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078853.749893:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961543808:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769078853.750396:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078853.750402:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078853.750405:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078853.750410:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4536 00000100:00080000:3.0:1769078853.750417:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078853.750423:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078853.750432:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078853.750458:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961543936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078853.750533:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961543936:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078853.750805:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078853.750820:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078853/real 1769078853] req@ffff895e453d6680 x1855008961543936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078853.750834:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961543936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078853.750842:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078853.750846:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078853.750849:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961543936:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078853.936019:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078854.320124:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078854.959960:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078855.343864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078855.984492:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078856.371563:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078857.007960:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078857.392531:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078858.032022:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078858.415892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078858.864013:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078858.864023:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078858.864027:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078858.864037:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4541 00000100:00080000:2.0:1769078858.864049:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078858.864056:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078858.864096:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961544064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078858.864126:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078858.864130:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078858.864132:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078858.864136:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4541 00000100:00080000:2.0:1769078858.864139:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078858.864143:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078858.864151:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078858.864162:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961544192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078858.864172:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078858.864175:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078858.864176:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078858.864180:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4541 00000100:00080000:2.0:1769078858.864183:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078858.864186:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078858.864230:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078858.864243:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961544320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078858.864459:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961544064:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078858.864521:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078858.864575:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078858/real 1769078858] req@ffff895e44e28000 x1855008961544064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078858.864591:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961544064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078858.864600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078858.864604:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078858.864608:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961544064:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078858.864615:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961544192:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078858.864635:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078858.864646:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078858/real 1769078858] req@ffff895e44e28380 x1855008961544192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078858.864656:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961544192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078858.864663:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078858.864666:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078858.864669:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961544192:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078858.864675:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961544320:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078858.864693:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078858.864703:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078858/real 1769078858] req@ffff895e44e29880 x1855008961544320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078858.864727:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961544320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078858.864766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078858.864771:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078858.864774:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961544320:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078859.056164:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078859.439879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078860.084137:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078860.463897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078861.103987:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078861.487982:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078862.128164:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078862.511882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078863.151999:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078863.535920:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078863.983858:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078863.983869:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078863.983873:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078863.983883:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4546 00000100:00080000:2.0:1769078863.983902:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078863.983909:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078863.983947:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961544448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078863.983977:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078863.983981:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078863.983983:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078863.983987:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4546 00000100:00080000:2.0:1769078863.983990:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078863.983993:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078863.984003:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078863.984014:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961544576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078863.984024:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078863.984027:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078863.984028:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078863.984031:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4546 00000100:00080000:2.0:1769078863.984048:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078863.984053:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078863.984058:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078863.984068:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961544704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078863.984355:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961544448:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078863.984466:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961544576:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078863.984491:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961544704:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078864.175972:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078864.560364:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078865.204017:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078865.584045:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078866.224228:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078866.607928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769078867.078646:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769078867.078709:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769078867.078727:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769078867.078821:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769078867.078831:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769078867.078880:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078863/real 1769078867] req@ffff895e44e29880 x1855008961544448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078867.078907:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961544448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078867.078950:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078867.078956:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078867.078960:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961544448:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078867.078969:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078863/real 1769078867] req@ffff895e44e28380 x1855008961544576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078867.078983:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961544576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078867.078991:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078867.078994:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078867.078999:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961544576:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078867.079006:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078863/real 1769078867] req@ffff895e44e28000 x1855008961544704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078867.079015:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961544704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078867.079022:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078867.079025:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078867.079027:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961544704:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078867.247998:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078867.633421:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078868.272023:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078868.655870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078869.104510:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078869.104522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078869.104526:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078869.104535:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4551 00000100:00080000:0.0:1769078869.104546:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078869.104554:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078869.104591:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961544832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078869.104625:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078869.104629:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078869.104643:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078869.104648:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4551 00000100:00080000:0.0:1769078869.104652:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078869.104656:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078869.104667:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078869.104681:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961544960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078869.104691:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078869.104694:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078869.104696:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078869.104699:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4551 00000100:00080000:0.0:1769078869.104703:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078869.104706:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078869.104710:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078869.104720:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961545088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078869.104834:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961544832:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078869.104896:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078869.104938:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078869/real 1769078869] req@ffff895e4383ce00 x1855008961544832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078869.104956:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961544832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078869.104965:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078869.104969:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078869.104972:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961544832:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078869.104978:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961544960:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078869.104999:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078869.105011:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078869/real 1769078869] req@ffff895e4383f480 x1855008961544960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078869.105034:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961544960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078869.105042:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078869.105045:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078869.105048:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961544960:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078869.105053:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961545088:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078869.105071:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078869.105080:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078869/real 1769078869] req@ffff895e4383c000 x1855008961545088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078869.105090:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961545088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078869.105107:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078869.105110:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078869.105113:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961545088:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078869.296004:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078869.679944:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078870.320020:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078870.703951:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078871.344101:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078871.727962:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078872.368015:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078872.751874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078873.392148:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078873.775856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078874.225019:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078874.225030:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078874.225034:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078874.225043:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4556 00000100:00080000:3.0:1769078874.225053:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078874.225062:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078874.225103:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961545216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078874.225136:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078874.225140:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078874.225141:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078874.225145:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4556 00000100:00080000:3.0:1769078874.225149:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078874.225152:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078874.225161:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078874.225173:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961545344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078874.225197:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078874.225201:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078874.225202:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078874.225206:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4556 00000100:00080000:3.0:1769078874.225210:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078874.225214:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078874.225219:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078874.225231:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961545472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078874.225301:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961545216:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078874.225360:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078874.225410:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078874/real 1769078874] req@ffff895e453d4380 x1855008961545216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078874.225441:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961545216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078874.225449:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078874.225453:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078874.225456:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961545216:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078874.225462:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961545344:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078874.225482:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078874.225493:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078874/real 1769078874] req@ffff895e453d5180 x1855008961545344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078874.225504:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961545344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078874.225511:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078874.225514:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078874.225517:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961545344:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078874.225521:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961545472:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078874.225538:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078874.225547:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078874/real 1769078874] req@ffff895e453d5880 x1855008961545472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078874.225558:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961545472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078874.225575:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078874.225578:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078874.225581:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961545472:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078874.420691:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078874.801158:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078875.439944:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078875.823877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078876.464130:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078876.847964:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078877.489469:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078877.872021:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078878.512413:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078878.895938:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078879.344115:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078879.344127:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078879.344132:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078879.344142:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4561 00000100:00080000:0.0:1769078879.344162:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078879.344169:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078879.344207:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961545600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078879.344243:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078879.344247:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078879.344249:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078879.344252:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4561 00000100:00080000:0.0:1769078879.344255:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078879.344258:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078879.344267:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078879.344278:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961545728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078879.344309:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078879.344312:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078879.344314:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078879.344318:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4561 00000100:00080000:0.0:1769078879.344321:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078879.344325:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078879.344329:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078879.344340:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961545856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078879.344528:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961545600:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078879.344587:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078879.344637:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078879/real 1769078879] req@ffff895e4383ea00 x1855008961545600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078879.344654:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961545600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078879.344664:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078879.344668:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078879.344671:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961545600:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078879.344685:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961545728:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078879.344708:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078879.344720:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078879/real 1769078879] req@ffff895e4383dc00 x1855008961545728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078879.344732:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961545728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078879.344740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078879.344743:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078879.344772:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961545728:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078879.344779:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961545856:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078879.344799:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078879.344810:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078879/real 1769078879] req@ffff895e4383ca80 x1855008961545856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078879.344821:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961545856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078879.344828:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078879.344831:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078879.344834:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961545856:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078879.535986:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078879.920976:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078880.559987:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078880.943899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078881.583980:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078881.968951:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078882.608101:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078882.991865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078883.633130:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078884.016237:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078884.464080:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078884.464091:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078884.464096:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078884.464105:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4567 00000100:00080000:0.0:1769078884.464116:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078884.464123:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078884.464163:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961545984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078884.464212:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078884.464216:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078884.464218:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078884.464234:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4567 00000100:00080000:0.0:1769078884.464238:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078884.464241:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078884.464249:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078884.464261:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961546112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078884.464271:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078884.464273:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078884.464275:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078884.464279:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4567 00000100:00080000:0.0:1769078884.464282:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078884.464286:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078884.464290:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078884.464300:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961546240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078884.466975:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961545984:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078884.467051:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078884.467106:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078884/real 1769078884] req@ffff895e4383d500 x1855008961545984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078884.467124:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961545984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078884.467135:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078884.467139:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078884.467143:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961545984:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078884.467150:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961546112:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078884.467184:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078884.467198:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078884/real 1769078884] req@ffff895e4383f800 x1855008961546112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078884.467210:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961546112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078884.467218:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078884.467221:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078884.467224:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961546112:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078884.467230:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961546240:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078884.467248:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078884.467259:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078884/real 1769078884] req@ffff895e4383e300 x1855008961546240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078884.467284:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961546240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078884.467292:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078884.467295:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078884.467298:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961546240:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078884.656016:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078885.040311:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078885.680468:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078886.079311:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078886.703978:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078887.087947:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078887.728407:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078888.112011:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078888.752038:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078889.136322:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078889.585290:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078889.585300:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078889.585304:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078889.585314:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4572 00000100:00080000:3.0:1769078889.585324:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078889.585329:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078889.585361:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961546368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078889.585388:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078889.585391:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078889.585392:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078889.585395:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4572 00000100:00080000:3.0:1769078889.585397:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078889.585411:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078889.585418:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078889.585427:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961546496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078889.585433:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078889.585435:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078889.585436:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078889.585438:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4572 00000100:00080000:3.0:1769078889.585441:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078889.585443:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078889.585446:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078889.585450:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961546624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078889.585516:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961546368:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078889.585577:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078889.585607:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078889/real 1769078889] req@ffff895e453d4700 x1855008961546368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078889.585620:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961546368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078889.585630:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078889.585633:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078889.585635:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961546368:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078889.585640:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961546496:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078889.585668:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078889.585690:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078889/real 1769078889] req@ffff895e453d7100 x1855008961546496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078889.585702:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961546496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078889.585711:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078889.585714:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078889.585717:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961546496:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078889.585723:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961546624:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078889.585741:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078889.585774:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078889/real 1769078889] req@ffff895e453d7b80 x1855008961546624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078889.585785:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961546624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078889.585793:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078889.585796:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078889.585799:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961546624:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078889.776779:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078890.159970:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078890.800003:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078891.183878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078891.824566:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078892.207910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078892.848840:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078893.231882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078893.872788:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078894.256335:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078894.705075:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078894.705087:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078894.705092:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078894.705105:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4577 00000100:00080000:3.0:1769078894.705118:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078894.705126:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078894.705166:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961546752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078894.705217:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078894.705221:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078894.705223:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078894.705227:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4577 00000100:00080000:3.0:1769078894.705232:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078894.705236:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078894.705245:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078894.705261:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961546880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078894.705271:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078894.705273:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078894.705275:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078894.705278:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4577 00000100:00080000:3.0:1769078894.705281:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078894.705284:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078894.705289:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078894.705313:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961547008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078894.708527:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961546752:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078894.708623:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078894.708678:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078894/real 1769078894] req@ffff895e453d5500 x1855008961546752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078894.708708:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961546752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078894.708719:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078894.708723:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078894.708727:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961546752:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078894.708735:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961546880:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078894.708805:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078894.708817:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078894/real 1769078894] req@ffff895e453d4e00 x1855008961546880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078894.708830:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961546880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078894.708839:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078894.708842:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078894.708845:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961546880:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078894.708852:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961547008:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078894.708887:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078894.708900:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078894/real 1769078894] req@ffff895e453d4a80 x1855008961547008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078894.708912:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961547008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078894.708919:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078894.708922:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078894.708925:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961547008:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078894.895949:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078895.279894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078895.920145:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078896.303928:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078896.944535:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078897.331010:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078897.967972:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078898.351850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078898.992470:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078899.379822:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078899.824697:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078899.824731:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078899.824735:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078899.824835:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4582 00000100:00080000:3.0:1769078899.824845:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078899.824852:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078899.824885:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961547136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078899.824905:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078899.824909:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078899.824910:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078899.824914:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4582 00000100:00080000:3.0:1769078899.824918:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078899.824921:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078899.824929:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078899.824940:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961547264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078899.824949:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078899.824952:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078899.824953:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078899.824956:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4582 00000100:00080000:3.0:1769078899.824959:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078899.824962:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078899.824967:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078899.824975:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961547392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078899.826588:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961547136:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078899.826659:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078899.826696:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078899/real 1769078899] req@ffff895e453d7480 x1855008961547136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078899.826715:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961547136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078899.826724:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078899.826728:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078899.826732:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961547136:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078899.826740:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961547264:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078899.826787:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078899.826812:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078899/real 1769078899] req@ffff895e453d5f80 x1855008961547264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078899.826824:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961547264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078899.826833:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078899.826836:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078899.826839:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961547264:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078899.826845:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961547392:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078899.826865:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078899.826876:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078899/real 1769078899] req@ffff895e453d5c00 x1855008961547392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078899.826889:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961547392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078899.826896:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078899.826899:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078899.826903:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961547392:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078900.017720:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078900.400431:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078901.041989:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078901.423881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078902.066704:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078902.448565:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078903.088976:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078903.474092:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078904.112637:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078904.514809:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078904.945575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078904.945586:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078904.945590:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078904.945600:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4587 00000100:00080000:1.0:1769078904.945609:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078904.945616:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078904.945670:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961547520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078904.947473:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961547520:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078904.947532:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:1.0:1769078904.949472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078904.949478:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078904.949482:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078904.949488:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4587 00000100:00080000:1.0:1769078904.949495:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078904.949500:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078904.949511:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078904.949534:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961547648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078904.949629:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078904/real 1769078904] req@ffff895e4713f100 x1855008961547520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078904.949650:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961547520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078904.949661:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078904.949664:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078904.949668:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961547520:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078904.949691:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961547648:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078904.949722:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078904.949733:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078904/real 1769078904] req@ffff895e4713ea00 x1855008961547648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078904.949758:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961547648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078904.949767:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078904.949770:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078904.949773:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961547648:192.168.204.117@tcp:38:kworker.0 00000100:00080000:1.0:1769078904.951481:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078904.951487:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078904.951489:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078904.951495:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4587 00000100:00080000:1.0:1769078904.951502:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078904.951521:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078904.951529:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078904.951548:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961547776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078904.951634:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961547776:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078904.951670:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078904.951686:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078904/real 1769078904] req@ffff895e4713ce00 x1855008961547776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078904.951700:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961547776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078904.951721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078904.951724:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078904.951727:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961547776:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078905.187351:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078905.520532:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078906.159982:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078906.543997:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078907.185717:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078907.567929:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078908.208028:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078908.591883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078909.232118:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078909.616061:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078910.064132:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078910.064143:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078910.064148:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078910.064157:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4592 00000100:00080000:3.0:1769078910.064168:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078910.064200:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078910.064254:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961547904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078910.064290:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078910.064294:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078910.064295:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078910.064299:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4592 00000100:00080000:3.0:1769078910.064303:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078910.064307:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078910.064317:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078910.064341:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961548032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078910.064351:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078910.064354:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078910.064356:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078910.064360:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4592 00000100:00080000:3.0:1769078910.064363:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078910.064367:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078910.064389:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078910.064401:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961548160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078910.066840:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961547904:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078910.066913:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078910.066970:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078910/real 1769078910] req@ffff895e453d7800 x1855008961547904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078910.066990:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961547904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078910.067001:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078910.067005:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078910.067008:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961547904:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078910.067016:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961548032:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078910.067039:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078910.067051:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078910/real 1769078910] req@ffff895e453d5c00 x1855008961548032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078910.067062:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961548032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078910.067089:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078910.067092:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078910.067096:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961548032:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078910.067102:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961548160:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078910.067123:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078910.067133:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078910/real 1769078910] req@ffff895e453d5f80 x1855008961548160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078910.067144:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961548160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078910.067151:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078910.067154:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078910.067157:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961548160:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078910.255995:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078910.639891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078911.280082:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078911.664443:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078912.303965:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078912.687911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078913.327991:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078913.712805:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078914.352083:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078914.736020:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078915.183864:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078915.183875:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078915.183879:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078915.183888:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4597 00000100:00080000:3.0:1769078915.183898:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078915.183905:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078915.183938:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961548288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078915.185344:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961548288:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078915.185420:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769078915.186300:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078915.186305:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078915.186309:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078915.186315:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4597 00000100:00080000:3.0:1769078915.186321:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078915.186327:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078915.186337:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078915.186362:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961548416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078915.188570:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078915/real 1769078915] req@ffff895e453d7480 x1855008961548288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078915.188597:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961548288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078915.188626:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078915.188631:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078915.188635:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961548288:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078915.188676:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961548416:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078915.188733:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078915.188763:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078915/real 1769078915] req@ffff895e453d4a80 x1855008961548416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078915.188775:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961548416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078915.188783:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078915.188786:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078915.188790:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961548416:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769078915.188869:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078915.188874:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078915.188876:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078915.188881:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4597 00000100:00080000:3.0:1769078915.188892:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078915.188897:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078915.188907:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078915.188928:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961548544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078915.188997:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961548544:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078915.189019:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078915.189030:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078915/real 1769078915] req@ffff895e453d4e00 x1855008961548544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078915.189041:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961548544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078915.189048:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078915.189051:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078915.189055:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961548544:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078915.375991:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078915.759873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078916.402661:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078916.783869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078917.423976:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078917.808046:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078918.447969:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078918.832502:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078919.476160:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078919.855851:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769078920.306837:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078920.306849:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769078920.306853:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078920.306863:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4602 00000100:00080000:0.0:1769078920.306874:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078920.306881:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769078920.306921:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961548672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078920.306953:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078920.306958:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769078920.306959:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078920.306963:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4602 00000100:00080000:0.0:1769078920.306967:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078920.306970:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078920.306980:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078920.306992:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961548800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769078920.307002:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769078920.307005:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769078920.307007:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769078920.307027:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4602 00000100:00100000:3.0:1769078920.307227:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961548672:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078920.307292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078920.307333:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078920/real 1769078920] req@ffff895e4383e300 x1855008961548672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078920.307350:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961548672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078920.307361:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078920.307365:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078920.307369:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961548672:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078920.307376:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961548800:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078920.307399:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078920.307411:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078920/real 1769078920] req@ffff895e4383f800 x1855008961548800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078920.307423:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961548800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078920.307431:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078920.307434:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078920.307444:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961548800:192.168.204.117@tcp:38:kworker.0 00000100:00080000:0.0:1769078920.307508:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769078920.307515:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769078920.307521:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769078920.307533:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961548928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078920.307581:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961548928:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078920.307607:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078920.311294:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078920/real 1769078920] req@ffff895e4383d500 x1855008961548928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078920.311320:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961548928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078920.311330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078920.311335:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078920.311339:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961548928:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078920.496477:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078920.879858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078921.520112:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078921.903830:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078922.544277:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078922.928227:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078923.568222:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078923.951956:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078924.594520:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078924.976421:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078925.425306:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078925.425317:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078925.425322:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078925.425332:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4607 00000100:00080000:1.0:1769078925.425357:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078925.425366:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078925.425404:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961549056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078925.425484:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961549056:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078925.425556:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:1.0:1769078925.425590:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078925.425594:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078925.425596:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078925.425600:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4607 00000100:00080000:1.0:1769078925.425604:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078925.425608:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078925.425618:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078925.425630:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961549184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078925.425700:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078925/real 1769078925] req@ffff895e4713d500 x1855008961549056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078925.425719:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961549056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078925.425728:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078925.425733:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078925.425736:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961549056:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078925.425795:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961549184:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078925.425819:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078925.425848:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078925/real 1769078925] req@ffff895e4713f480 x1855008961549184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078925.425860:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961549184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078925.425868:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078925.425871:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078925.425874:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961549184:192.168.204.117@tcp:38:kworker.0 00000100:00080000:1.0:1769078925.428080:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078925.428089:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078925.428093:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078925.428103:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4607 00000100:00080000:1.0:1769078925.428112:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078925.428118:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078925.428147:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078925.428173:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961549312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078925.428577:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961549312:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078925.428633:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078925.428658:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078925/real 1769078925] req@ffff895e4713ca80 x1855008961549312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078925.428674:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961549312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078925.428684:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078925.428687:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078925.428691:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961549312:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078925.615995:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078926.000268:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078926.642330:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078927.023957:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078927.665513:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078928.048269:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078928.688886:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078929.072167:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078929.712155:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078930.096840:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078930.543892:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078930.543903:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078930.543907:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078930.543917:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4613 00000100:00080000:3.0:1769078930.543929:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078930.543935:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078930.543990:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961549440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078930.544066:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961549440:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078930.544223:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078930.544228:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078930.544230:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078930.544234:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4613 00000100:00080000:3.0:1769078930.544238:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078930.544242:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078930.544251:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078930.544264:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961549568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078930.544780:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961549568:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769078930.545280:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078930.545284:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078930.545286:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078930.545291:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4613 00000100:00080000:3.0:1769078930.545621:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078930.545628:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078930.545635:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078930.545653:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961549696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078930.545706:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961549696:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078930.736102:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078931.120143:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078931.764264:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078932.143886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078932.786348:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078933.167899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769078933.619358:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769078933.620727:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769078933.620765:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078933.620816:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769078933.620829:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769078933.621197:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078930/real 1769078933] req@ffff895e453d7100 x1855008961549696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078933.621227:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961549696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078933.621257:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078933.621264:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078933.621269:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961549696:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078933.621277:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078930/real 1769078933] req@ffff895e453d7b80 x1855008961549568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078933.621290:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961549568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078933.621309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078933.621313:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078933.621316:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961549568:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078933.621322:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078930/real 1769078933] req@ffff895e453d5500 x1855008961549440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078933.621335:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961549440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078933.621341:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078933.621344:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078933.621347:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961549440:192.168.204.117@tcp:250:kworker.0 00010000:00010000:0.0:1769078933.808250:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078934.191981:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078934.832064:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078935.215924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078935.665283:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078935.665295:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078935.665299:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078935.665309:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4618 00000100:00080000:1.0:1769078935.665321:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078935.665328:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078935.665367:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961549824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078935.665402:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078935.665406:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078935.665407:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078935.665482:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4618 00000100:00080000:1.0:1769078935.665486:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078935.665490:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078935.665500:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078935.665513:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961549952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078935.665523:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078935.665526:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078935.665527:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078935.665530:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4618 00000100:00080000:1.0:1769078935.665534:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078935.665538:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078935.665542:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078935.665552:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961550080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078935.667059:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961549824:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078935.667165:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078935.667205:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078935/real 1769078935] req@ffff895e4713f800 x1855008961549824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078935.667250:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961549824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078935.667261:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078935.667266:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078935.667269:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961549824:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078935.667276:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961549952:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078935.667299:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078935.667312:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078935/real 1769078935] req@ffff895e4713dc00 x1855008961549952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078935.667322:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961549952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078935.667329:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078935.667332:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078935.667334:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961549952:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078935.667340:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961550080:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078935.667356:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078935.667364:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078935/real 1769078935] req@ffff895e4713df80 x1855008961550080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078935.667373:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961550080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078935.667393:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078935.667396:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078935.667399:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961550080:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078935.856228:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078936.239997:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078936.880006:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078937.263913:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078937.904710:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078938.287921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078938.928149:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078939.311902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078939.952902:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078940.335844:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078940.787153:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078940.787164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078940.787169:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078940.787180:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4623 00000100:00080000:2.0:1769078940.787190:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078940.787197:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078940.787232:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961550208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078940.787267:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078940.787270:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078940.787272:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078940.787276:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4623 00000100:00080000:2.0:1769078940.787279:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078940.787282:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078940.787305:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078940.787315:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961550336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078940.787324:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078940.787328:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078940.787329:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078940.787332:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4623 00000100:00080000:2.0:1769078940.787335:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078940.787339:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078940.787343:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078940.787351:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961550464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078940.788589:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961550208:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078940.788673:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078940.788731:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078940/real 1769078940] req@ffff895e44e2aa00 x1855008961550208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078940.788788:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961550208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078940.788799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078940.788803:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078940.788806:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961550208:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078940.788822:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961550336:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078940.788845:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078940.788857:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078940/real 1769078940] req@ffff895e44e2a300 x1855008961550336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078940.788869:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961550336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078940.788877:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078940.788880:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078940.788883:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961550336:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078940.788902:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961550464:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078940.788921:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078940.788931:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078940/real 1769078940] req@ffff895e44e2b100 x1855008961550464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078940.788941:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961550464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078940.788949:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078940.788952:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078940.788955:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961550464:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078940.976005:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078941.360238:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078942.000205:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078942.383927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078943.024698:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078943.409154:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078944.052877:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078944.431868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078945.073431:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078945.455893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769078945.904176:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078945.904187:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769078945.904190:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078945.904200:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4628 00000100:00080000:2.0:1769078945.904211:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078945.904218:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769078945.904256:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961550592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078945.904292:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078945.904296:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769078945.904297:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078945.904301:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4628 00000100:00080000:2.0:1769078945.904320:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078945.904325:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078945.904334:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078945.904347:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961550720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769078945.904357:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769078945.904366:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769078945.904368:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769078945.904372:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4628 00000100:00080000:2.0:1769078945.904375:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769078945.904379:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769078945.904384:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769078945.904394:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961550848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078945.904491:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961550592:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078945.904553:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078945.904601:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078945/real 1769078945] req@ffff895e44e2b480 x1855008961550592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078945.904620:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961550592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078945.904630:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078945.904634:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078945.904637:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961550592:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078945.904644:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961550720:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078945.904666:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078945.904679:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078945/real 1769078945] req@ffff895e44e28e00 x1855008961550720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078945.904700:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961550720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078945.904709:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078945.904712:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078945.904731:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961550720:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078945.904738:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961550848:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078945.904795:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078945.904806:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078945/real 1769078945] req@ffff895e44e29f80 x1855008961550848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078945.904818:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961550848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078945.904826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078945.904829:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078945.904831:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961550848:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078946.098330:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078946.479919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078947.120118:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078947.504371:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078948.149105:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078948.528025:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078949.168804:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078949.551871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078950.192402:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078950.575879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078951.024435:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078951.024448:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078951.024453:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078951.024462:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4633 00000100:00080000:1.0:1769078951.024474:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078951.024481:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078951.024538:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961550976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078951.024563:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078951.024566:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078951.024568:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078951.024572:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4633 00000100:00080000:1.0:1769078951.024576:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078951.024579:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078951.024588:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078951.024599:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961551104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078951.024630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078951.024633:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078951.024644:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078951.024648:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4633 00000100:00080000:1.0:1769078951.024652:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078951.024657:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078951.024661:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078951.024673:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961551232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078951.025391:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961550976:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078951.025457:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078951.025494:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078951/real 1769078951] req@ffff895e4713d180 x1855008961550976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078951.025510:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961550976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078951.025544:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078951.025548:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078951.025552:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961550976:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078951.025559:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961551104:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078951.025582:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078951.025594:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078951/real 1769078951] req@ffff895e4713c380 x1855008961551104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078951.025606:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961551104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078951.025613:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078951.025615:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078951.025618:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961551104:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078951.025624:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961551232:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078951.025652:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078951.025662:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078951/real 1769078951] req@ffff895e4713df80 x1855008961551232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078951.025672:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961551232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078951.025678:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078951.025681:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078951.025683:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961551232:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078951.216474:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078951.599874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078952.242371:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078952.623898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078953.264124:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078953.648358:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078954.287948:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078954.672067:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078955.311956:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078955.696368:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078956.143925:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078956.143937:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078956.143941:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078956.143952:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4638 00000100:00080000:1.0:1769078956.143963:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078956.143970:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078956.144324:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961551360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078956.144360:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078956.144378:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078956.144380:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078956.144384:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4638 00000100:00080000:1.0:1769078956.144389:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078956.144393:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078956.144402:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078956.144417:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961551488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078956.144426:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078956.144429:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078956.144431:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078956.144434:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4638 00000100:00080000:1.0:1769078956.144437:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078956.144441:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078956.144445:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078956.144454:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961551616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769078956.144573:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961551360:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769078956.144644:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078956.144680:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078956/real 1769078956] req@ffff895e4713dc00 x1855008961551360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078956.144697:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961551360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078956.144706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078956.144710:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078956.144713:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961551360:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769078956.145160:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961551488:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078956.145187:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078956.145200:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078956/real 1769078956] req@ffff895e4713f800 x1855008961551488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078956.145213:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961551488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078956.145221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078956.145224:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078956.145228:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961551488:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769078956.145233:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961551616:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769078956.145250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769078956.145260:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078956/real 1769078956] req@ffff895e4713ca80 x1855008961551616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769078956.145278:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961551616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769078956.145285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769078956.145288:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769078956.145291:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961551616:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078956.346947:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078956.720881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078957.359964:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078957.745981:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078958.384016:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078958.767997:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078959.407934:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078959.793228:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078960.435404:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078960.815910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078961.264973:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078961.264994:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078961.264999:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078961.265010:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4643 00000100:00080000:3.0:1769078961.265021:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078961.265029:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078961.265068:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961551744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078961.265102:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078961.265106:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078961.265108:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078961.265112:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4643 00000100:00080000:3.0:1769078961.265116:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078961.265120:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078961.265129:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078961.265140:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961551872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078961.265171:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078961.265174:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078961.265176:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078961.265180:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4643 00000100:00080000:3.0:1769078961.265184:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078961.265188:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078961.265192:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078961.265203:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961552000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078961.265296:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961551744:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078961.265369:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078961.265420:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078961/real 1769078961] req@ffff895e453d5500 x1855008961551744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078961.265439:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961551744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078961.265449:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078961.265453:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078961.265457:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961551744:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078961.265472:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961551872:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078961.265496:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078961.265508:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078961/real 1769078961] req@ffff895e453d7b80 x1855008961551872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078961.265518:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961551872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078961.265526:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078961.265536:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078961.265539:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961551872:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078961.265545:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961552000:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078961.265572:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078961.265583:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078961/real 1769078961] req@ffff895e453d7100 x1855008961552000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078961.265594:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961552000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078961.265600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078961.265603:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078961.265605:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961552000:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078961.455978:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078961.839955:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078962.483215:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078962.863881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078963.503963:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078963.887884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078964.527945:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078964.914206:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078965.553842:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078965.937690:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078966.384732:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078966.384774:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078966.384779:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078966.384789:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4648 00000100:00080000:3.0:1769078966.384800:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078966.384807:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078966.384844:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961552128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078966.384865:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078966.384868:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078966.384870:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078966.384874:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4648 00000100:00080000:3.0:1769078966.384877:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078966.384880:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078966.384915:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078966.384929:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961552256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078966.384939:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078966.384943:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078966.384944:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078966.384948:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4648 00000100:00080000:3.0:1769078966.384952:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078966.384956:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078966.384960:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078966.384969:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961552384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078966.385154:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961552128:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078966.385228:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078966.385264:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078966/real 1769078966] req@ffff895e453d4700 x1855008961552128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078966.385281:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961552128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078966.385290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078966.385293:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078966.385304:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961552128:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078966.385312:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961552256:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078966.385332:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078966.385344:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078966/real 1769078966] req@ffff895e453d5880 x1855008961552256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078966.385355:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961552256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078966.385363:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078966.385366:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078966.385369:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961552256:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078966.385374:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961552384:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078966.385404:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078966.385415:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078966/real 1769078966] req@ffff895e453d5180 x1855008961552384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078966.385426:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961552384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078966.385433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078966.385436:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078966.385438:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961552384:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078966.577940:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078966.960265:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078967.601233:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078967.983879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078968.623970:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078969.007894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078969.648428:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078970.031879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078970.672380:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769078971.055922:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078971.503862:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078971.503883:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078971.503887:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078971.503897:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4654 00000100:00080000:1.0:1769078971.503907:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078971.503915:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078971.503957:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961552512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078971.503991:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078971.503994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078971.503996:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078971.504000:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4654 00000100:00080000:1.0:1769078971.504004:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078971.504008:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078971.504017:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078971.504046:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961552640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078971.504055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078971.504058:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078971.504060:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078971.504064:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4654 00000100:00080000:1.0:1769078971.504068:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078971.504072:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078971.504077:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078971.504086:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961552768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078971.504176:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961552512:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078971.504238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078971.504276:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078971/real 1769078971] req@ffff895e4713f480 x1855008961552512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078971.504305:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961552512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769078996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078971.504314:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078971.504318:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078971.504322:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961552512:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078971.504329:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961552640:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078971.504350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078971.504362:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078971/real 1769078971] req@ffff895e4713d500 x1855008961552640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078971.504373:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961552640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078971.504380:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078971.504383:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078971.504386:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961552640:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078971.504391:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961552768:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078971.504408:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078971.504417:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078971/real 1769078971] req@ffff895e4713ce00 x1855008961552768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078971.504436:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961552768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769078996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078971.504443:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078971.504447:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078971.504450:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961552768:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078971.696083:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078972.082574:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078972.719975:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078973.103867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078973.744226:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078974.128430:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078974.768568:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078975.151864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078975.795520:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078976.175880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078976.623894:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078976.623905:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078976.623909:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078976.623918:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4659 00000100:00080000:1.0:1769078976.624107:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078976.624198:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078976.624245:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961552896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078976.624387:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961552896:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078976.624444:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:1.0:1769078976.624472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078976.624476:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078976.624478:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078976.624496:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4659 00000100:00080000:1.0:1769078976.624500:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078976.624504:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078976.624513:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078976.624526:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961553024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078976.628384:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078976/real 1769078976] req@ffff895e4713ea00 x1855008961552896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078976.628411:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961552896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078976.628421:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078976.628426:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078976.628430:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961552896:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078976.628466:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961553024:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078976.628521:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078976.628537:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078976/real 1769078976] req@ffff895e4713f100 x1855008961553024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078976.628582:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961553024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078976.628590:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078976.628593:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078976.628597:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961553024:192.168.204.117@tcp:38:kworker.0 00000100:00080000:1.0:1769078976.637698:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078976.637709:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078976.637712:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078976.637722:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4659 00000100:00080000:1.0:1769078976.637732:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078976.637740:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078976.637768:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078976.637805:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961553152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078976.637978:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961553152:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078976.638037:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078976.638079:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078976/real 1769078976] req@ffff895e4713e300 x1855008961553152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078976.638096:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961553152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078976.638106:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078976.638119:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078976.638123:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961553152:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078976.816773:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078977.200787:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078977.839952:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078978.223892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078978.864381:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078979.253727:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078979.888022:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078980.271880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078980.911979:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078981.295862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769078981.744038:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078981.744051:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769078981.744055:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078981.744065:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4664 00000100:00080000:3.0:1769078981.744095:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078981.744103:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769078981.744152:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961553280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078981.744183:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078981.744187:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769078981.744189:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078981.744193:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4664 00000100:00080000:3.0:1769078981.744196:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078981.744200:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078981.744209:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078981.744233:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961553408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769078981.744242:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769078981.744246:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769078981.744248:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769078981.744253:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4664 00000100:00100000:0.0:1769078981.744256:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961553280:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769078981.744258:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769078981.744264:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769078981.744270:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769078981.744281:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961553536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769078981.744326:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078981.744376:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078981/real 1769078981] req@ffff895e453d4380 x1855008961553280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078981.744410:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961553280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078981.744420:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078981.744426:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078981.744429:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961553280:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078981.744437:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961553408:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078981.744461:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078981.744473:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078981/real 1769078981] req@ffff895e453d6680 x1855008961553408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078981.744486:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961553408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078981.744494:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078981.744497:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078981.744501:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961553408:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078981.744533:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961553536:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078981.744554:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078981.744565:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078981/real 1769078981] req@ffff895e453d6300 x1855008961553536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078981.744577:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961553536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078981.744595:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078981.744599:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078981.744996:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961553536:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078981.938332:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078982.321561:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078982.965867:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078983.343846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078983.983989:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078984.367870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078985.008231:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078985.393184:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078986.032102:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078986.415930:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078986.865536:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078986.865548:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078986.865553:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078986.865563:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4669 00000100:00080000:1.0:1769078986.865574:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078986.865581:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078986.865635:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961553664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078986.865663:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078986.865665:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078986.865667:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078986.865669:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4669 00000100:00080000:1.0:1769078986.865672:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078986.865675:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078986.865683:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078986.865691:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961553792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078986.865696:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078986.865699:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078986.865700:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078986.865702:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4669 00000100:00080000:1.0:1769078986.865703:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078986.865705:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078986.865707:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078986.865712:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961553920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078986.865854:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961553664:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078986.865910:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078986.865946:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078986/real 1769078986] req@ffff895e4713d880 x1855008961553664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078986.865975:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961553664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078986.865987:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078986.865990:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078986.865993:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961553664:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078986.865998:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961553792:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078986.866016:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078986.866025:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078986/real 1769078986] req@ffff895e4713ed80 x1855008961553792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078986.866044:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961553792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078986.866068:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078986.866073:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078986.866076:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961553792:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078986.866082:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961553920:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078986.866113:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078986.866124:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078986/real 1769078986] req@ffff895e4713c000 x1855008961553920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078986.866133:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961553920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078986.866140:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078986.866435:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078986.866443:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961553920:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078987.056591:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078987.439868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078988.079987:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078988.464608:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078989.103977:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078989.488380:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078990.128050:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078990.513152:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078991.152435:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078991.538011:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078991.983907:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078991.983955:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078991.983960:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078991.983972:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4674 00000100:00080000:1.0:1769078991.983987:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078991.983996:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078991.984041:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961554048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078991.984077:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078991.984081:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078991.984082:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078991.984086:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4674 00000100:00080000:1.0:1769078991.984090:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078991.984094:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078991.984103:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078991.984114:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961554176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078991.984123:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078991.984126:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078991.984128:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078991.984131:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4674 00000100:00080000:1.0:1769078991.984134:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078991.984137:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078991.984142:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078991.984151:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961554304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078991.984329:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961554048:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769078991.984392:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078991.984432:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078991/real 1769078991] req@ffff895e4713fb80 x1855008961554048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078991.984448:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961554048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078991.984469:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078991.984473:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078991.984477:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961554048:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078991.984484:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961554176:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078991.984505:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078991.984518:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078991/real 1769078991] req@ffff895e4713c700 x1855008961554176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078991.984528:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961554176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078991.984536:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078991.984538:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078991.984541:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961554176:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078991.984546:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961554304:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769078991.984563:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769078991.984573:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078991/real 1769078991] req@ffff895e4713c000 x1855008961554304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769078991.984585:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961554304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769078991.984592:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769078991.984595:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769078991.984598:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961554304:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769078992.175982:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078992.561215:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078993.199974:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078993.583968:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769078994.224850:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078994.609818:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078995.251231:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078995.631880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078996.272988:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078996.656638:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769078997.103884:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078997.103895:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769078997.103901:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078997.103911:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4679 00000100:00080000:1.0:1769078997.103922:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078997.103930:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769078997.103970:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961554432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078997.104003:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078997.104006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769078997.104008:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078997.104013:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4679 00000100:00080000:1.0:1769078997.104016:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078997.104019:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078997.104028:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078997.104039:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961554560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769078997.104049:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769078997.104052:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769078997.104053:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769078997.104056:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4679 00000100:00080000:1.0:1769078997.104059:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769078997.104077:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769078997.104082:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769078997.104092:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961554688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769078997.104343:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961554432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769078997.104485:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961554560:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769078997.104512:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961554688:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769078997.297388:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078997.679878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078998.325941:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078998.703816:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769078999.344101:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769078999.728049:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769079000.183875:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769079000.183940:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769079000.183965:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769079000.184018:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769079000.184028:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769079000.185165:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078997/real 1769079000] req@ffff895e4713ed80 x1855008961554432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079000.185206:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961554432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079000.185220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079000.185226:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079000.185231:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961554432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079000.185249:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078997/real 1769079000] req@ffff895e4713d880 x1855008961554560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079000.185263:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961554560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079000.185281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079000.185285:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079000.185318:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961554560:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079000.185327:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769078997/real 1769079000] req@ffff895e4713e300 x1855008961554688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079000.185341:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961554688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079000.185348:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079000.185352:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079000.185355:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961554688:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079000.368787:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079000.752182:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079001.392909:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079001.776870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079002.223960:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079002.223976:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079002.223981:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079002.223993:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4684 00000100:00080000:1.0:1769079002.224008:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079002.224016:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079002.224062:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961554816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079002.224093:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079002.224097:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079002.224099:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079002.224103:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4684 00000100:00080000:1.0:1769079002.224106:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079002.224111:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079002.224124:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079002.224137:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961554944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079002.224146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769079002.224165:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961554816:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769079002.224172:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079002.224173:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079002.224177:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4684 00000100:00080000:1.0:1769079002.224181:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079002.224185:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079002.224190:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079002.224201:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961555072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769079002.224258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079002.224309:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079002/real 1769079002] req@ffff895e4713f100 x1855008961554816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079002.224328:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961554816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079002.224339:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079002.224344:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079002.224349:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961554816:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079002.224357:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961554944:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079002.224379:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079002.224392:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079002/real 1769079002] req@ffff895e4713ea00 x1855008961554944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079002.224418:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961554944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079002.224426:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079002.224429:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079002.224433:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961554944:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079002.224463:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961555072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079002.224485:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079002.224496:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079002/real 1769079002] req@ffff895e4713ce00 x1855008961555072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079002.224507:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961555072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079002.224514:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079002.224517:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079002.224520:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961555072:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079002.417499:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079002.800833:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079003.441975:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079003.825469:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079004.464076:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079004.848208:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079005.488900:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079005.872071:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079006.513570:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079006.895985:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079007.352295:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079007.352310:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079007.352316:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079007.352329:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4689 00000100:00080000:2.0:1769079007.352346:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079007.352387:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079007.352435:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961555200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079007.352561:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961555200:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079007.352648:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:2.0:1769079007.354482:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079007.354494:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079007.354499:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079007.354510:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4689 00000100:00080000:2.0:1769079007.354524:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079007.354532:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079007.354635:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079007.354680:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961555328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079007.361185:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079007/real 1769079007] req@ffff895e44e29500 x1855008961555200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079007.361249:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961555200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079007.361261:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079007.361267:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079007.361271:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961555200:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079007.361319:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961555328:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079007.361403:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079007.361424:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079007/real 1769079007] req@ffff895e453d6d80 x1855008961555328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079007.361436:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961555328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079007.361445:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079007.361448:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079007.361451:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961555328:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769079007.363828:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079007.363844:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079007.363849:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079007.363965:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4689 00000100:00080000:3.0:1769079007.363981:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079007.363989:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079007.364008:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079007.364119:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961555456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079007.364697:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961555456:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079007.364837:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079007.369489:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079007/real 1769079007] req@ffff895e453d6d80 x1855008961555456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079007.369529:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961555456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079007.369575:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079007.369582:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079007.369587:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961555456:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079007.536245:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079007.922386:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079008.560084:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079008.943924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079009.584920:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079009.969967:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079010.608074:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079010.992291:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079011.632189:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079012.016094:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079012.464886:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079012.464903:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079012.464908:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079012.464921:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4695 00000100:00080000:1.0:1769079012.464935:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079012.464945:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079012.464994:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961555584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079012.465036:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079012.465040:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079012.465042:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079012.465047:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4695 00000100:00080000:1.0:1769079012.465051:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079012.465054:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079012.465085:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079012.465099:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961555712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079012.465109:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079012.465113:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079012.465115:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079012.465119:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4695 00000100:00080000:1.0:1769079012.465123:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079012.465128:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079012.465133:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079012.465143:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961555840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079012.466036:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961555584:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079012.466144:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079012.466198:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079012/real 1769079012] req@ffff895e4713d500 x1855008961555584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079012.466219:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961555584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079012.466230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079012.466234:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079012.466238:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961555584:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079012.466246:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961555712:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079012.466269:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079012.466282:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079012/real 1769079012] req@ffff895e4713f480 x1855008961555712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079012.466294:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961555712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079012.466323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079012.466327:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079012.466331:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961555712:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079012.466337:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961555840:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079012.466360:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079012.466371:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079012/real 1769079012] req@ffff895e4713ca80 x1855008961555840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079012.466381:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961555840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079012.466388:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079012.466392:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079012.466395:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961555840:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079012.656145:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079013.040207:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079013.680043:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079014.067983:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079014.704129:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079015.087855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079015.728097:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079016.111930:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079016.752401:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079017.136020:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079017.584165:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079017.584178:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079017.584182:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079017.584193:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4700 00000100:00080000:1.0:1769079017.584205:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079017.584212:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079017.584256:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961555968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079017.584290:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079017.584294:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079017.584312:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079017.584317:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4700 00000100:00080000:1.0:1769079017.584322:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079017.584326:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079017.584335:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079017.584346:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961556096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079017.584355:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079017.584359:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079017.584360:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079017.584363:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4700 00000100:00080000:1.0:1769079017.584366:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079017.584370:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079017.584374:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079017.584383:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961556224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079017.584615:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961555968:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079017.584674:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079017.584713:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079017/real 1769079017] req@ffff895e4713f800 x1855008961555968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079017.584728:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961555968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079017.584743:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079017.584764:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079017.584767:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961555968:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079017.584773:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961556096:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079017.584790:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079017.584798:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079017/real 1769079017] req@ffff895e4713dc00 x1855008961556096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079017.584805:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961556096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079017.584810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079017.584812:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079017.584814:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961556096:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079017.584817:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961556224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079017.584827:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079017.584832:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079017/real 1769079017] req@ffff895e4713df80 x1855008961556224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079017.584838:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961556224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079017.584843:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079017.584844:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079017.584850:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961556224:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079017.776232:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079018.159927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079018.800316:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079019.183883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079019.824660:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079020.208584:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079020.848089:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079021.232068:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079021.872221:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079022.255947:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079022.703863:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079022.703874:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079022.703878:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079022.703888:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4705 00000100:00080000:1.0:1769079022.703899:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079022.703906:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079022.703975:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961556352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079022.704006:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079022.704010:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079022.704012:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079022.704016:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4705 00000100:00080000:1.0:1769079022.704020:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079022.704024:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079022.704033:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079022.704045:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961556480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079022.704053:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079022.704056:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079022.704058:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079022.704061:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4705 00000100:00080000:1.0:1769079022.704064:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079022.704067:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079022.704072:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079022.704082:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961556608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079022.704178:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961556352:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079022.704236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079022.704270:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079022/real 1769079022] req@ffff895e4713c380 x1855008961556352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079022.704285:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961556352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079022.704295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079022.704298:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079022.704301:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961556352:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079022.704308:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961556480:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079022.704329:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079022.704341:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079022/real 1769079022] req@ffff895e4713d180 x1855008961556480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079022.704352:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961556480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079022.704360:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079022.704363:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079022.704365:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961556480:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079022.704371:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961556608:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079022.704397:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079022.704408:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079022/real 1769079022] req@ffff895e4713df80 x1855008961556608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079022.704419:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961556608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079022.704425:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079022.704462:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079022.704466:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961556608:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079022.896731:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079023.280660:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079023.920109:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079024.303892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079024.944496:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079025.329035:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079025.968468:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079026.351831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079026.992603:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079027.376017:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079027.826096:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079027.826108:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079027.826131:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079027.826140:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4710 00000100:00080000:1.0:1769079027.826151:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079027.826158:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079027.826190:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961556736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079027.826223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079027.826227:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079027.826229:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079027.826232:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4710 00000100:00080000:1.0:1769079027.826236:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079027.826239:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079027.826248:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079027.826258:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961556864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079027.826267:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079027.826270:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079027.826272:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079027.826275:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4710 00000100:00080000:1.0:1769079027.826278:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079027.826282:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079027.826286:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079027.826296:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961556992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079027.826389:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961556736:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079027.826450:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079027.826487:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079027/real 1769079027] req@ffff895e4713dc00 x1855008961556736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079027.826504:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961556736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079027.826514:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079027.826518:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079027.826522:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961556736:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079027.826528:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961556864:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079027.826562:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079027.826576:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079027/real 1769079027] req@ffff895e4713f800 x1855008961556864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079027.826587:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961556864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079027.826593:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079027.826597:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079027.826600:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961556864:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079027.826606:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961556992:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079027.826638:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079027.826649:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079027/real 1769079027] req@ffff895e4713ca80 x1855008961556992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079027.826659:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961556992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079027.826666:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079027.826669:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079027.826672:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961556992:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079028.018312:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079028.399870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079029.041540:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079029.424372:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079030.063973:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079030.447875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079031.090621:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079031.471867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079032.112000:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079032.496109:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079032.949062:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079032.949074:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079032.949078:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079032.949088:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4715 00000100:00080000:2.0:1769079032.949099:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079032.949107:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079032.949145:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961557120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079032.949178:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079032.949182:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079032.949183:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079032.949187:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4715 00000100:00080000:2.0:1769079032.949190:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079032.949194:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079032.949203:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079032.949233:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961557248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079032.949243:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079032.949246:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079032.949247:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079032.949251:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4715 00000100:00080000:2.0:1769079032.949263:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079032.949268:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079032.949273:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079032.949284:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961557376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079032.949442:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961557120:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079032.950026:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079032.950073:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079032/real 1769079032] req@ffff895e44e28a80 x1855008961557120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079032.950091:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961557120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079032.950101:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079032.950105:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079032.950109:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961557120:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079032.950118:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961557248:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079032.950143:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079032.950154:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079032/real 1769079032] req@ffff895e44e29180 x1855008961557248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079032.950166:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961557248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079032.950174:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079032.950177:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079032.950180:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961557248:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079032.950186:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961557376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079032.950205:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079032.950215:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079032/real 1769079032] req@ffff895e44e2bb80 x1855008961557376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079032.950225:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961557376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079032.950243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079032.950246:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079032.950250:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961557376:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079033.136532:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079033.520217:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079034.163110:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079034.543893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079035.184018:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079035.569058:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079036.208028:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079036.592576:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079037.231915:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079037.615937:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079038.064908:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079038.064918:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079038.064922:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079038.064932:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4720 00000100:00080000:2.0:1769079038.064943:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079038.064949:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079038.064994:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961557504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079038.065036:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079038.065040:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079038.065042:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079038.065045:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4720 00000100:00080000:2.0:1769079038.065049:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079038.065068:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079038.065078:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079038.065090:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961557632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079038.065099:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079038.065102:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079038.065104:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079038.065107:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4720 00000100:00080000:2.0:1769079038.065110:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079038.065114:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079038.065118:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079038.065127:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961557760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079038.066233:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961557504:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079038.066333:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079038.066380:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079038/real 1769079038] req@ffff895e44e2b800 x1855008961557504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079038.066397:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961557504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079038.066406:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079038.066410:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079038.066413:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961557504:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079038.066420:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961557632:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079038.066439:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079038.066452:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079038/real 1769079038] req@ffff895e44e29c00 x1855008961557632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079038.066462:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961557632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079038.066488:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079038.066491:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079038.066495:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961557632:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079038.066501:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961557760:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079038.066522:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079038.066531:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079038/real 1769079038] req@ffff895e44e2bb80 x1855008961557760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079038.066541:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961557760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079038.066547:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079038.066550:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079038.066553:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961557760:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079038.258088:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079038.639918:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079039.282081:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079039.663885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079040.313377:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079040.688036:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079041.327965:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079041.712366:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079042.352074:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079042.737739:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079043.184040:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079043.184069:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079043.184077:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079043.184115:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4725 00000100:00080000:2.0:1769079043.184146:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079043.184159:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079043.184222:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961557888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079043.184332:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961557888:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079043.184430:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:2.0:1769079043.184472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079043.184477:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079043.184501:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079043.184506:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4725 00000100:00080000:2.0:1769079043.184511:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079043.184516:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079043.184530:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079043.184546:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961558016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079043.197515:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079043/real 1769079043] req@ffff895e44e29180 x1855008961557888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079043.197550:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961557888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079043.197584:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079043.197590:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079043.197595:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961557888:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079043.197637:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961558016:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079043.197710:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079043.197727:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079043/real 1769079043] req@ffff895e44e28a80 x1855008961558016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079043.197739:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961558016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079043.197773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079043.197777:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079043.197780:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961558016:192.168.204.117@tcp:38:kworker.0 00000100:00080000:2.0:1769079043.201816:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079043.201830:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079043.201836:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079043.201849:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4725 00000100:00080000:2.0:1769079043.201864:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079043.201872:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079043.201889:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079043.201935:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961558144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079043.203827:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961558144:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079043.203937:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079043.203979:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079043/real 1769079043] req@ffff895e44e29500 x1855008961558144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079043.204001:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961558144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079043.204014:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079043.204019:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079043.204023:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961558144:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079043.379996:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079043.759943:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079044.401967:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079044.783979:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079045.424121:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079045.811853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079046.448830:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079046.835206:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079047.472201:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079047.856402:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079048.309225:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079048.309240:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079048.309246:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079048.309269:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4730 00000100:00080000:1.0:1769079048.309285:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079048.309293:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079048.309342:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961558272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079048.309387:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079048.309391:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079048.309393:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079048.309398:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4730 00000100:00080000:1.0:1769079048.309402:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079048.309406:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079048.309417:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079048.309431:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961558400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079048.309441:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079048.309462:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079048.309464:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079048.309468:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4730 00000100:00080000:1.0:1769079048.309471:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079048.309475:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079048.309480:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079048.309491:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961558528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079048.309678:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961558272:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079048.309816:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079048.309865:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079048/real 1769079048] req@ffff895e4713f480 x1855008961558272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079048.309883:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961558272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079048.309894:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079048.309898:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079048.309902:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961558272:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079048.309909:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961558400:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079048.309932:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079048.309945:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079048/real 1769079048] req@ffff895e4713d500 x1855008961558400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079048.309955:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961558400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079048.309963:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079048.309967:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079048.309969:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961558400:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079048.309975:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961558528:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079048.310017:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079048.310028:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079048/real 1769079048] req@ffff895e4713ce00 x1855008961558528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079048.310041:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961558528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079048.310049:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079048.310052:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079048.310055:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961558528:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079048.496248:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079048.879952:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079049.520175:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079049.905181:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079050.544843:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079050.927882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079051.568014:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079051.952865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079052.592878:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079052.976243:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079053.424017:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079053.424029:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079053.424034:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079053.424043:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4735 00000100:00080000:1.0:1769079053.424054:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079053.424061:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079053.424100:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961558656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079053.424133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079053.424137:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079053.424138:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079053.424142:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4735 00000100:00080000:1.0:1769079053.424146:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079053.424150:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079053.424176:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079053.424199:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961558784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079053.424209:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079053.424212:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079053.424214:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079053.424218:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4735 00000100:00080000:1.0:1769079053.424221:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079053.424225:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079053.424229:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079053.424237:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961558912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079053.424339:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961558656:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079053.424411:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079053.424450:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079053/real 1769079053] req@ffff895e4713ea00 x1855008961558656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079053.424468:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961558656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079053.424478:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079053.424482:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079053.424486:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961558656:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079053.424492:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961558784:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079053.424513:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079053.424534:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079053/real 1769079053] req@ffff895e4713f100 x1855008961558784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079053.424545:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961558784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079053.424553:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079053.424556:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079053.424559:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961558784:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079053.424565:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961558912:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079053.424584:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079053.424594:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079053/real 1769079053] req@ffff895e4713e300 x1855008961558912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079053.424605:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961558912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079053.424612:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079053.424615:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079053.424618:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961558912:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079053.616604:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079054.000598:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079054.640951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079055.028101:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079055.664067:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079056.055133:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079056.689797:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079057.071924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079057.712508:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079058.097216:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079058.543963:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079058.543974:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079058.543978:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079058.543988:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4741 00000100:00080000:1.0:1769079058.544000:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079058.544006:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079058.544043:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961559040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079058.544075:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079058.544078:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079058.544095:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079058.544099:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4741 00000100:00080000:1.0:1769079058.544103:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079058.544107:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079058.544116:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079058.544129:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961559168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079058.544138:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079058.544141:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079058.544143:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079058.544146:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4741 00000100:00080000:1.0:1769079058.544149:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079058.544153:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079058.544157:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079058.544166:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961559296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079058.544493:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961559040:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079058.544556:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079058.544606:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079058/real 1769079058] req@ffff895e4713d880 x1855008961559040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079058.545164:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961559040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079058.545176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079058.545191:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079058.545195:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961559040:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079058.545203:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961559168:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079058.545231:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079058.545375:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079058/real 1769079058] req@ffff895e4713ed80 x1855008961559168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079058.545462:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961559168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079058.545471:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079058.545475:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079058.545478:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961559168:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079058.545484:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961559296:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079058.545651:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079058.545662:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079058/real 1769079058] req@ffff895e4713c000 x1855008961559296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079058.545673:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961559296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079058.545693:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079058.545696:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079058.545699:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961559296:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079058.737048:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079059.120417:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079059.759993:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079060.143875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079060.784176:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079061.168309:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079061.808133:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079062.191939:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079062.832070:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079063.215925:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079063.664058:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079063.664087:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079063.664091:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079063.664102:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4746 00000100:00080000:3.0:1769079063.664124:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079063.664132:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079063.664176:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961559424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079063.664203:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079063.664206:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079063.664208:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079063.664212:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4746 00000100:00080000:3.0:1769079063.664215:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079063.664218:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079063.664227:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079063.664238:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961559552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079063.664359:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961559424:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769079063.664525:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079063.664528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079063.664530:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079063.664534:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4746 00000100:00080000:3.0:1769079063.664538:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079063.664542:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079063.664547:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079063.664559:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961559680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079063.664639:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961559552:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079063.664669:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961559680:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079063.856348:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079064.239953:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079064.881502:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079065.264057:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079065.904049:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079066.288740:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769079066.736209:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769079066.736256:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769079066.736275:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769079066.736318:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769079066.736328:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769079066.736421:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079063/real 1769079066] req@ffff895e453d6680 x1855008961559552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079066.736448:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961559552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079066.736460:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079066.736465:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079066.736469:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961559552:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079066.736477:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079063/real 1769079066] req@ffff895e453d4380 x1855008961559680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079066.736489:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961559680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079066.736511:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079066.736516:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079066.736519:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961559680:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079066.736526:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079063/real 1769079066] req@ffff895e453d6300 x1855008961559424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079066.736538:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961559424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079066.736545:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079066.736548:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079066.736551:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961559424:192.168.204.117@tcp:250:kworker.0 00010000:00010000:0.0:1769079066.931538:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079067.311883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079067.952430:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079068.337496:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079068.783870:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079068.783882:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079068.783886:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079068.783896:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4751 00000100:00080000:3.0:1769079068.783907:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079068.783913:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079068.783948:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961559808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079068.783986:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079068.783989:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079068.783991:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079068.783994:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4751 00000100:00080000:3.0:1769079068.783998:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079068.784001:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079068.784009:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079068.784022:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961559936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079068.784030:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079068.784033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079068.784047:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079068.784051:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4751 00000100:00080000:3.0:1769079068.784055:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079068.784058:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079068.784064:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079068.784075:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961560064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079068.784305:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961559808:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079068.784364:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079068.784399:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079068/real 1769079068] req@ffff895e453d5180 x1855008961559808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079068.784414:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961559808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079068.784422:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079068.784426:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079068.784429:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961559808:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079068.784435:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961559936:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079068.784464:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079068.784475:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079068/real 1769079068] req@ffff895e453d5880 x1855008961559936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079068.784488:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961559936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079068.784495:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079068.784497:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079068.784500:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961559936:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079068.784506:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961560064:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079068.784523:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079068.784532:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079068/real 1769079068] req@ffff895e453d4700 x1855008961560064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079068.784543:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961560064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079068.784551:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079068.784554:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079068.784557:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961560064:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079068.976565:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079069.360477:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079070.001022:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079070.384905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079071.024024:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079071.408130:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079072.052104:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079072.431946:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079073.083331:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079073.455931:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079073.903968:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079073.903978:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079073.903983:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079073.903992:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4756 00000100:00080000:2.0:1769079073.904003:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079073.904010:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079073.904045:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961560192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079073.904077:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079073.904081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079073.904082:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079073.904086:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4756 00000100:00080000:2.0:1769079073.904090:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079073.904094:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079073.904103:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079073.904115:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961560320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079073.904124:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079073.904127:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079073.904129:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079073.904132:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4756 00000100:00080000:2.0:1769079073.904135:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079073.904139:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079073.904143:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079073.904169:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961560448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079073.904273:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961560192:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079073.904339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079073.904373:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079073/real 1769079073] req@ffff895e44e29f80 x1855008961560192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079073.904389:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961560192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079073.904399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079073.904403:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079073.904407:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961560192:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079073.904413:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961560320:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079073.904432:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079073.904443:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079073/real 1769079073] req@ffff895e44e28e00 x1855008961560320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079073.904455:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961560320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079073.904463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079073.904466:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079073.904469:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961560320:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079073.904474:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961560448:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079073.904491:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079073.904500:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079073/real 1769079073] req@ffff895e44e2b480 x1855008961560448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079073.904523:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961560448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079073.904531:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079073.904533:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079073.904536:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961560448:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079074.097003:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079074.479857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079075.119970:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079075.504052:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079076.145137:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079076.528357:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079077.168567:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079077.551933:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079078.193645:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079078.575876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079079.026156:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079079.026167:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079079.026172:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079079.026183:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4761 00000100:00080000:0.0:1769079079.026194:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079079.026202:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079079.026239:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961560576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079079.026270:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079079.026274:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079079.026276:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079079.026279:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4761 00000100:00080000:0.0:1769079079.026283:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079079.026286:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079079.026296:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079079.026318:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961560576:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079079.026328:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961560704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079079.026340:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079079.026343:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079079.026345:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079079.026349:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4761 00000100:00080000:0.0:1769079079.026353:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079079.026357:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079079.026361:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079079.026370:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961560832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769079079.026554:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079079.026591:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079079/real 1769079079] req@ffff895e4383ca80 x1855008961560576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079079.026609:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961560576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079079.026618:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079079.026623:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079079.026626:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961560576:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079079.026658:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961560704:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079079.026681:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079079.026692:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079079/real 1769079079] req@ffff895e4383dc00 x1855008961560704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079079.026705:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961560704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079079.026712:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079079.026716:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079079.026719:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961560704:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079079.026725:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961560832:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079079.026742:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079079.026782:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079079/real 1769079079] req@ffff895e4383ea00 x1855008961560832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079079.026793:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961560832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079079.026800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079079.026803:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079079.026807:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961560832:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079079.216180:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079079.600387:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079080.240287:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079080.623879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079081.265249:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079081.647968:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079082.293228:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079082.671867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079083.314095:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079083.696316:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079084.143965:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079084.143975:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079084.143979:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079084.143988:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4766 00000100:00080000:0.0:1769079084.143999:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079084.144006:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079084.144042:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961560960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079084.144072:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079084.144076:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079084.144091:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079084.144095:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4766 00000100:00080000:0.0:1769079084.144100:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079084.144103:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079084.144112:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079084.144124:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961561088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079084.144133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079084.144136:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079084.144138:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079084.144141:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4766 00000100:00080000:0.0:1769079084.144144:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079084.144147:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079084.144151:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079084.144162:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961561216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079084.144335:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961560960:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079084.144413:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079084.144835:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079084/real 1769079084] req@ffff895e4383c000 x1855008961560960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079084.144854:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961560960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079084.144864:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079084.144868:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079084.144872:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961560960:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079084.144880:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961561088:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079084.144904:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079084.144917:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079084/real 1769079084] req@ffff895e4383f480 x1855008961561088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079084.144928:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961561088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079084.144936:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079084.144939:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079084.144942:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961561088:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079084.144948:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961561216:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079084.144966:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079084.144976:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079084/real 1769079084] req@ffff895e4383ce00 x1855008961561216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079084.144996:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961561216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079084.145004:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079084.145007:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079084.145010:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961561216:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079084.336024:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079084.719941:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079085.360425:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079085.746249:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079086.385638:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079086.767940:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079087.408875:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079087.792607:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079088.432031:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079088.815817:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079089.266112:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079089.266193:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079089.266197:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079089.266207:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4771 00000100:00080000:3.0:1769079089.266218:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079089.266226:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079089.266267:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961561344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079089.266302:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079089.266316:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079089.266318:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079089.266322:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4771 00000100:00080000:3.0:1769079089.266326:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079089.266347:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079089.266356:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079089.266369:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961561472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079089.266378:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079089.266381:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079089.266383:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079089.266386:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4771 00000100:00080000:3.0:1769079089.266391:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079089.266394:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079089.266398:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079089.266406:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961561600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079089.266512:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961561344:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079089.266582:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079089.266617:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079089/real 1769079089] req@ffff895e453d7100 x1855008961561344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079089.266643:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961561344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079089.266652:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079089.266656:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079089.266660:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961561344:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079089.266667:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961561472:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079089.266688:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079089.266698:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079089/real 1769079089] req@ffff895e453d7b80 x1855008961561472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079089.266708:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961561472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079089.266714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079089.266716:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079089.266719:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961561472:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079089.266723:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961561600:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079089.266739:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079089.266781:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079089/real 1769079089] req@ffff895e453d5500 x1855008961561600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079089.266793:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961561600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079089.266813:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079089.266816:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079089.266820:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961561600:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079089.458974:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079089.839874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079090.480469:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079090.866085:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079091.503993:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079091.887907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079092.528374:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079092.911868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079093.553078:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079093.935920:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079094.386022:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079094.386035:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079094.386042:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079094.386053:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4776 00000100:00080000:2.0:1769079094.386065:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079094.386074:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079094.386120:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961561728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079094.386228:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961561728:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079094.386299:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:2.0:1769079094.386334:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079094.386338:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079094.386340:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079094.386344:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4776 00000100:00080000:2.0:1769079094.386348:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079094.386352:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079094.386361:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079094.386374:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961561856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079094.392449:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079094/real 1769079094] req@ffff895e44e2b100 x1855008961561728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079094.392481:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961561728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079094.392493:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079094.392498:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079094.392504:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961561728:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079094.392546:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961561856:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079094.392612:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079094.392629:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079094/real 1769079094] req@ffff895e44e2a300 x1855008961561856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079094.392641:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961561856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079094.392669:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079094.392673:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079094.392677:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961561856:192.168.204.117@tcp:38:kworker.0 00000100:00080000:2.0:1769079094.395775:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079094.395787:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079094.395791:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079094.395802:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4776 00000100:00080000:2.0:1769079094.395821:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079094.395829:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079094.395864:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079094.395898:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961561984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079094.399087:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961561984:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079094.399172:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079094.399441:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079094/real 1769079094] req@ffff895e44e2aa00 x1855008961561984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079094.399461:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961561984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079094.399471:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079094.399476:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079094.399480:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961561984:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079094.576046:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079094.960260:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079095.600000:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079095.984629:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079096.627429:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079097.007959:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079097.648243:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079098.031829:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079098.672304:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079099.055895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079099.503885:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079099.503896:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079099.503900:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079099.503935:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4782 00000100:00080000:3.0:1769079099.503946:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079099.503952:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079099.503991:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961562112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079099.504068:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961562112:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079099.504130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769079099.504261:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079099.504265:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079099.504266:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079099.504271:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4782 00000100:00080000:3.0:1769079099.504275:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079099.504279:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079099.504289:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079099.504300:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961562240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079099.504442:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079099/real 1769079099] req@ffff895e453d4e00 x1855008961562112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079099.504459:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961562112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079099.504481:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079099.504485:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079099.504489:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961562112:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079099.504511:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961562240:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079099.504535:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079099.504547:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079099/real 1769079099] req@ffff895e453d4a80 x1855008961562240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079099.504557:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961562240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079099.504564:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079099.504566:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079099.504569:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961562240:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769079099.504895:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079099.504900:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079099.504902:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079099.504906:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4782 00000100:00080000:3.0:1769079099.504910:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079099.505234:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079099.505241:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079099.505258:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961562368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079099.505335:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961562368:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079099.505363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079099.505376:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079099/real 1769079099] req@ffff895e453d7480 x1855008961562368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079099.505388:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961562368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079099.505395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079099.505398:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079099.505402:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961562368:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079099.696605:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079100.079848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079100.720207:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079101.105447:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079101.744368:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079102.128036:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079102.767889:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079103.152133:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079103.792816:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079104.175856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079104.624645:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079104.624657:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079104.624661:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079104.624671:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4787 00000100:00080000:1.0:1769079104.624683:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079104.624690:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079104.624740:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961562496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079104.624796:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079104.624799:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079104.624801:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079104.624805:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4787 00000100:00080000:1.0:1769079104.624809:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079104.624813:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079104.624821:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079104.624833:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961562624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079104.624841:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079104.624844:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079104.624846:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079104.624849:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4787 00000100:00080000:1.0:1769079104.624852:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079104.624856:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079104.624861:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079104.624871:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961562752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079104.624984:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961562496:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079104.625056:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079104.625091:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079104/real 1769079104] req@ffff895e4713c700 x1855008961562496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079104.625109:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961562496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079104.625119:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079104.625124:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079104.625127:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961562496:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079104.625135:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961562624:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079104.625157:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079104.625169:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079104/real 1769079104] req@ffff895e4713fb80 x1855008961562624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079104.625181:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961562624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079104.625188:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079104.625191:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079104.625194:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961562624:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079104.625209:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961562752:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079104.625230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079104.625240:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079104/real 1769079104] req@ffff895e4713c000 x1855008961562752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079104.625251:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961562752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079104.625258:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079104.625261:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079104.625264:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961562752:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079104.817171:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079105.199845:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079105.839927:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079106.225203:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079106.867860:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079107.247869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079107.888228:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079108.271984:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079108.912483:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079109.295963:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079109.746128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079109.746140:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079109.746144:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079109.746153:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4792 00000100:00080000:2.0:1769079109.746165:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079109.746171:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079109.746208:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961562880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079109.746239:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079109.746242:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079109.746258:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079109.746263:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4792 00000100:00080000:2.0:1769079109.746267:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079109.746270:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079109.746279:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079109.746291:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961563008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079109.746301:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079109.746304:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079109.746306:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079109.746309:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4792 00000100:00080000:2.0:1769079109.746312:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079109.746315:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079109.746319:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079109.746328:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961563136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079109.746410:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961562880:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079109.746468:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079109.746500:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079109/real 1769079109] req@ffff895e44e28000 x1855008961562880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079109.746516:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961562880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079109.746525:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079109.746529:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079109.746532:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961562880:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079109.746539:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961563008:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079109.746569:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079109.746580:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079109/real 1769079109] req@ffff895e44e28380 x1855008961563008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079109.746591:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961563008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079109.746599:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079109.746602:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079109.746605:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961563008:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079109.746610:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961563136:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079109.746627:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079109.746637:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079109/real 1769079109] req@ffff895e44e29880 x1855008961563136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079109.746647:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961563136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079109.746673:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079109.746676:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079109.746679:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961563136:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079109.953400:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079110.320022:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079110.960785:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079111.347808:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079111.984231:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079112.367857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079113.009605:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079113.392651:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079114.032379:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079114.415881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079114.868118:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079114.868132:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079114.868136:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079114.868145:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4797 00000100:00080000:2.0:1769079114.868157:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079114.868166:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079114.868228:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961563264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079114.868269:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079114.868273:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079114.868275:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079114.868279:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4797 00000100:00080000:2.0:1769079114.868283:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079114.868288:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079114.868309:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079114.868322:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961563392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079114.868331:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079114.868334:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079114.868336:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079114.868339:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4797 00000100:00080000:2.0:1769079114.868343:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079114.868346:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079114.868351:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079114.868359:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961563520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079114.868833:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961563264:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079114.868920:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079114.868967:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079114/real 1769079114] req@ffff895e44e29880 x1855008961563264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079114.868984:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961563264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079114.868994:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079114.868999:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079114.869003:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961563264:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079114.869010:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961563392:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079114.869031:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079114.869043:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079114/real 1769079114] req@ffff895e44e28380 x1855008961563392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079114.869054:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961563392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079114.869062:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079114.869065:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079114.869068:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961563392:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079114.869073:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961563520:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079114.869088:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079114.869097:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079114/real 1769079114] req@ffff895e44e28000 x1855008961563520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079114.869107:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961563520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079114.869125:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079114.869129:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079114.869134:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961563520:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079115.057340:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079115.443280:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079116.079941:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079116.463933:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079117.105274:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079117.487871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079118.128533:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079118.511879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079119.155737:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079119.535898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079119.990214:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079119.990226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079119.990232:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079119.990264:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4802 00000100:00080000:1.0:1769079119.990276:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079119.990283:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079119.990318:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961563648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079119.990347:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079119.990350:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079119.990352:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079119.990355:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4802 00000100:00080000:1.0:1769079119.990359:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079119.990363:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079119.990371:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079119.990382:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961563776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079119.990391:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079119.990394:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079119.990396:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079119.990400:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4802 00000100:00080000:1.0:1769079119.990403:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079119.990408:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079119.990412:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079119.990427:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961563904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079119.990522:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961563648:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079119.990599:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079119.990653:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079119/real 1769079119] req@ffff895e4713ed80 x1855008961563648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079119.990672:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961563648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079119.990682:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079119.990687:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079119.990691:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961563648:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079119.990699:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961563776:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079119.990720:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079119.990732:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079119/real 1769079119] req@ffff895e4713d880 x1855008961563776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079119.990742:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961563776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079119.990818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079119.990822:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079119.990825:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961563776:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079119.990830:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961563904:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079119.990850:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079119.990861:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079119/real 1769079119] req@ffff895e4713e300 x1855008961563904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079119.990871:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961563904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079119.990879:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079119.990881:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079119.990884:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961563904:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079120.177044:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079120.561559:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079121.202012:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079121.583926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079122.224017:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079122.607903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079123.248095:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079123.631902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079124.279588:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079124.656175:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079125.104036:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079125.104065:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079125.104070:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079125.104080:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4807 00000100:00080000:2.0:1769079125.104091:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079125.104098:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079125.104137:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961564032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079125.104170:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079125.104174:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079125.104176:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079125.104179:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4807 00000100:00080000:2.0:1769079125.104183:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079125.104187:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079125.104196:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079125.104207:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961564160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079125.104216:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079125.104220:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079125.104221:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079125.104225:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4807 00000100:00080000:2.0:1769079125.104228:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079125.104232:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079125.104236:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079125.104246:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961564288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079125.104350:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961564032:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079125.104411:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079125.104449:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079125/real 1769079125] req@ffff895e44e2aa00 x1855008961564032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079125.104467:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961564032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079125.104489:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079125.104493:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079125.104497:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961564032:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079125.104504:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961564160:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079125.104528:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079125.104540:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079125/real 1769079125] req@ffff895e44e2a300 x1855008961564160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079125.104552:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961564160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079125.104560:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079125.104563:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079125.104566:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961564160:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079125.104572:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961564288:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079125.104590:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079125.104601:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079125/real 1769079125] req@ffff895e44e2b100 x1855008961564288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079125.104612:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961564288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079125.104620:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079125.104623:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079125.104626:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961564288:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079125.296786:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079125.679884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079126.320417:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079126.703879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079127.348494:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079127.727875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079128.369359:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079128.751897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079129.391962:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079129.775879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079130.223916:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079130.223926:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079130.223930:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079130.223940:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4812 00000100:00080000:2.0:1769079130.223951:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079130.223959:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079130.224001:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961564416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079130.224047:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079130.224051:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079130.224053:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079130.224057:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4812 00000100:00080000:2.0:1769079130.224060:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079130.224064:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079130.224073:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079130.224084:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961564544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079130.224093:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079130.224096:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079130.224098:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079130.224101:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4812 00000100:00080000:2.0:1769079130.224118:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079130.224123:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079130.224128:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079130.224140:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961564672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079130.224420:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961564416:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079130.224523:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961564544:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079130.224548:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961564672:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079130.416270:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079130.799892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079131.439969:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079131.823874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079132.470685:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079132.848226:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769079133.296933:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769079133.296989:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769079133.297008:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079133.297050:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079133.297060:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769079133.297254:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079130/real 1769079133] req@ffff895e44e2b480 x1855008961564416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079133.297280:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961564416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079133.297292:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079133.297298:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079133.297302:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961564416:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079133.297310:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079130/real 1769079133] req@ffff895e44e28e00 x1855008961564544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079133.297322:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961564544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079133.297330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079133.297333:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079133.297336:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961564544:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079133.297360:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079130/real 1769079133] req@ffff895e44e29f80 x1855008961564672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079133.297373:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961564672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079133.297379:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079133.297383:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079133.297385:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961564672:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079133.488048:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079133.872439:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079134.513991:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079134.895953:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079135.343901:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079135.343912:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079135.343916:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079135.343925:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4817 00000100:00080000:3.0:1769079135.343937:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079135.343944:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079135.343985:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961564800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079135.344040:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079135.344043:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079135.344045:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079135.344050:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4817 00000100:00080000:3.0:1769079135.344054:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079135.344058:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079135.344067:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079135.344082:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961564928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079135.344092:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079135.344095:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079135.344097:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079135.344100:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4817 00000100:00080000:3.0:1769079135.344103:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079135.344107:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079135.344110:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079135.344119:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961565056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079135.344297:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961564800:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079135.344355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079135.344404:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079135/real 1769079135] req@ffff895e453d5f80 x1855008961564800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079135.344659:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961564800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079135.344670:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079135.344674:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079135.344678:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961564800:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079135.344686:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961564928:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079135.344724:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079135.344738:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079135/real 1769079135] req@ffff895e453d5c00 x1855008961564928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079135.348479:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961564928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079135.348493:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079135.350096:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079135.350105:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961564928:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079135.350120:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961565056:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079135.350202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079135.350234:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079135/real 1769079135] req@ffff895e453d7800 x1855008961565056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079135.350251:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961565056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079135.350260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079135.350264:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079135.350267:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961565056:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079135.536888:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079135.921576:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079136.559984:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079136.946650:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079137.585619:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079137.967909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079138.611973:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079138.992777:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079139.632189:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079140.015938:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079140.464950:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079140.464963:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079140.464968:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079140.464977:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4823 00000100:00080000:0.0:1769079140.464989:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079140.465012:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079140.465044:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961565184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079140.465077:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079140.465080:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079140.465081:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079140.465085:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4823 00000100:00080000:0.0:1769079140.465089:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079140.465092:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079140.465101:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079140.465113:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961565312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079140.465121:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079140.465124:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079140.465126:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079140.465129:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4823 00000100:00080000:0.0:1769079140.465132:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079140.465135:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079140.465139:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079140.465148:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961565440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079140.465340:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961565184:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079140.465401:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079140.465463:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079140/real 1769079140] req@ffff895e4383fb80 x1855008961565184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079140.465494:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961565184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079140.465502:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079140.465506:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079140.465509:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961565184:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079140.465516:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961565312:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079140.465538:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079140.465548:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079140/real 1769079140] req@ffff895e4383c380 x1855008961565312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079140.465560:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961565312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079140.465567:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079140.465570:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079140.465573:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961565312:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079140.465578:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961565440:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079140.465595:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079140.465605:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079140/real 1769079140] req@ffff895e4383f100 x1855008961565440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079140.465627:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961565440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079140.465645:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079140.465649:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079140.465652:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961565440:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079140.658973:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079141.042128:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079141.680119:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079142.069794:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079142.704052:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079143.087874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079143.727963:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079144.112258:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079144.756465:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079145.137712:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079145.583938:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079145.583949:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079145.583954:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079145.584674:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4828 00000100:00080000:2.0:1769079145.584689:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079145.584697:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079145.584737:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961565568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079145.584849:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961565568:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079145.584928:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:2.0:1769079145.584961:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079145.584974:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079145.584977:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079145.584981:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4828 00000100:00080000:2.0:1769079145.584985:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079145.584988:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079145.584997:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079145.585025:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961565696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079145.601633:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079145/real 1769079145] req@ffff895e44e29500 x1855008961565568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079145.601660:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961565568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079145.601669:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079145.601674:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079145.601678:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961565568:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079145.601712:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961565696:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079145.601789:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079145.601806:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079145/real 1769079145] req@ffff895e44e28a80 x1855008961565696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079145.601828:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961565696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079145.601836:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079145.601839:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079145.601842:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961565696:192.168.204.117@tcp:38:kworker.0 00000100:00080000:2.0:1769079145.603432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079145.603440:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079145.603443:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079145.603451:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4828 00000100:00080000:2.0:1769079145.603460:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079145.603467:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079145.603480:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079145.603511:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961565824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079145.604503:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961565824:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079145.604577:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079145.604601:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079145/real 1769079145] req@ffff895e44e29180 x1855008961565824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079145.604618:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961565824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079145.604627:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079145.604631:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079145.604635:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961565824:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079145.776057:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079146.161587:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079146.799967:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079147.184114:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079147.825970:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079148.211097:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079148.851100:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079149.232332:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079149.871932:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079150.255869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079150.703997:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079150.704009:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079150.704013:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079150.704023:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4833 00000100:00080000:3.0:1769079150.704035:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079150.704043:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079150.704082:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961565952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079150.704118:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079150.704129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079150.704131:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079150.704149:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4833 00000100:00080000:3.0:1769079150.704154:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079150.704158:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079150.704167:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079150.704180:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961566080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079150.704190:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079150.704193:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079150.704195:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079150.704198:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4833 00000100:00080000:3.0:1769079150.704201:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079150.704205:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079150.704209:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079150.704218:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961566208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079150.704322:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961565952:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079150.704390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079150.704428:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079150/real 1769079150] req@ffff895e453d7800 x1855008961565952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079150.704443:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961565952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079150.704463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079150.704467:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079150.704471:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961565952:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079150.704477:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961566080:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079150.704496:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079150.704506:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079150/real 1769079150] req@ffff895e453d5c00 x1855008961566080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079150.704516:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961566080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079150.704535:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079150.704538:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079150.704541:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961566080:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079150.704555:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961566208:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079150.704574:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079150.704583:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079150/real 1769079150] req@ffff895e453d5f80 x1855008961566208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079150.704594:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961566208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079150.704601:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079150.704604:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079150.704607:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961566208:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079150.899703:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079151.284561:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079151.922878:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079152.303872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079152.943980:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079153.327877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079153.967975:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079154.351971:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079154.992022:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079155.375990:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079155.826228:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079155.826239:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079155.826243:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079155.826253:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4838 00000100:00080000:2.0:1769079155.826263:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079155.826270:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079155.826306:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961566336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079155.826337:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079155.826340:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079155.826356:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079155.826361:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4838 00000100:00080000:2.0:1769079155.826365:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079155.826368:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079155.826378:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079155.826390:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961566464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079155.826399:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079155.826402:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079155.826404:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079155.826407:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4838 00000100:00080000:2.0:1769079155.826409:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079155.826413:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079155.826417:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079155.826426:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961566592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079155.826640:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961566336:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079155.826699:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079155.826774:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079155/real 1769079155] req@ffff895e44e2bb80 x1855008961566336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079155.826793:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961566336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079155.826802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079155.826806:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079155.826810:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961566336:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079155.826817:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961566464:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079155.826845:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079155.826857:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079155/real 1769079155] req@ffff895e44e29c00 x1855008961566464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079155.826868:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961566464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079155.826875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079155.826878:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079155.826881:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961566464:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079155.826887:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961566592:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079155.826905:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079155.826914:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079155/real 1769079155] req@ffff895e44e2b800 x1855008961566592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079155.826924:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961566592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079155.826931:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079155.826933:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079155.826936:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961566592:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079156.016041:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079156.399874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079157.040042:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079157.424511:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079158.064479:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079158.450257:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079159.088171:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079159.471946:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079160.112183:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079160.495897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079160.944200:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079160.944374:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079160.944379:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079160.944390:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4843 00000100:00080000:2.0:1769079160.944403:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079160.944410:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079160.944464:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961566720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079160.944499:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079160.944503:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079160.944505:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079160.944508:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4843 00000100:00080000:2.0:1769079160.944512:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079160.944516:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079160.944525:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079160.944537:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961566848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079160.944546:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079160.944551:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769079160.944553:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961566720:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769079160.944553:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079160.944561:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4843 00000100:00080000:2.0:1769079160.944565:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079160.944568:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079160.944573:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079160.944580:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961566976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769079160.944738:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079160.944804:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079160/real 1769079160] req@ffff895e44e2b800 x1855008961566720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079160.944822:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961566720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079160.944832:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079160.944836:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079160.944840:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961566720:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079160.944865:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961566848:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079160.944887:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079160.944898:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079160/real 1769079160] req@ffff895e44e29c00 x1855008961566848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079160.944909:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961566848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079160.944918:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079160.944921:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079160.944924:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961566848:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079160.944930:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961566976:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079160.944947:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079160.944956:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079160/real 1769079160] req@ffff895e44e2bb80 x1855008961566976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079160.944967:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961566976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079160.945393:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079160.945397:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079160.945401:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961566976:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079161.138452:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079161.520120:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079162.159973:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079162.544149:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079163.184667:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079163.567965:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079164.212903:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079164.595712:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079165.237638:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079165.619711:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079166.064418:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079166.064445:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079166.064449:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079166.064459:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4848 00000100:00080000:3.0:1769079166.064471:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079166.064478:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079166.064516:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961567104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079166.064550:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079166.064554:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079166.064555:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079166.064559:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4848 00000100:00080000:3.0:1769079166.064563:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079166.064567:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079166.064576:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079166.064590:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961567232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079166.064599:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079166.064602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079166.064604:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079166.064607:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4848 00000100:00080000:3.0:1769079166.064610:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079166.064613:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079166.064617:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079166.064626:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961567360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079166.064742:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961567104:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079166.064857:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079166.064891:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079166/real 1769079166] req@ffff895e453d7480 x1855008961567104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079166.064907:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961567104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079166.064916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079166.064920:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079166.064923:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961567104:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079166.064931:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961567232:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079166.064952:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079166.064963:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079166/real 1769079166] req@ffff895e453d4a80 x1855008961567232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079166.064975:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961567232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079166.064994:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079166.064997:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079166.065000:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961567232:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079166.065006:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961567360:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079166.065038:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079166.065048:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079166/real 1769079166] req@ffff895e453d4e00 x1855008961567360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079166.065071:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961567360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079166.065298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079166.065304:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079166.065308:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961567360:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079166.256281:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079166.640078:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079167.280358:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079167.663926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079168.306390:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079168.689484:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079169.328074:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079169.711899:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079170.357180:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079170.736004:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079171.187638:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079171.187667:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079171.187671:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079171.187682:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4853 00000100:00080000:2.0:1769079171.187693:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079171.187710:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079171.187743:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961567488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079171.187792:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079171.187796:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079171.187798:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079171.187801:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4853 00000100:00080000:2.0:1769079171.187805:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079171.187809:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079171.187817:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079171.187829:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961567616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079171.187837:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079171.187840:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079171.187841:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079171.187845:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4853 00000100:00080000:2.0:1769079171.187848:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079171.187851:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079171.187855:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079171.187864:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961567744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079171.188021:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961567488:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079171.188083:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079171.188121:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079171/real 1769079171] req@ffff895e44e29180 x1855008961567488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079171.188136:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961567488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079171.188156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079171.188159:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079171.188163:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961567488:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079171.188169:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961567616:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079171.188203:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079171.188215:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079171/real 1769079171] req@ffff895e44e28a80 x1855008961567616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079171.188226:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961567616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079171.188232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079171.188235:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079171.188238:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961567616:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079171.188243:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961567744:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079171.188261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079171.188273:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079171/real 1769079171] req@ffff895e44e29500 x1855008961567744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079171.188283:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961567744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079171.188290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079171.188293:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079171.188296:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961567744:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079171.375996:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079171.760033:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079172.402194:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079172.783912:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079173.424060:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079173.807872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079174.449260:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079174.831897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079175.472312:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079175.855920:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079176.304941:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079176.304953:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079176.304958:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079176.304968:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4858 00000100:00080000:1.0:1769079176.304979:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079176.304986:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079176.305023:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961567872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079176.305050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079176.305054:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079176.305056:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079176.305075:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4858 00000100:00080000:1.0:1769079176.305080:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079176.305086:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079176.305087:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961567872:192.168.204.117@tcp:250:kworker.0 00000008:00100000:1.0:1769079176.305096:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079176.305112:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961568000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079176.305124:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079176.305127:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079176.305129:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079176.305132:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4858 00000100:00080000:1.0:1769079176.305136:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079176.305139:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079176.305144:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079176.305154:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961568128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769079176.305259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079176.305296:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079176/real 1769079176] req@ffff895e4713f100 x1855008961567872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079176.305313:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961567872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079176.305323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079176.305327:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079176.305330:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961567872:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079176.305352:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961568000:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079176.305372:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079176.305383:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079176/real 1769079176] req@ffff895e4713ea00 x1855008961568000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079176.305393:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961568000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079176.305401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079176.305404:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079176.305407:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961568000:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079176.305412:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961568128:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079176.305437:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079176.305446:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079176/real 1769079176] req@ffff895e4713ce00 x1855008961568128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079176.305457:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961568128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079176.305465:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079176.305468:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079176.305471:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961568128:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079176.495955:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079176.880053:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079177.520011:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079177.904094:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079178.544249:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079178.928201:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079179.568204:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079179.951900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079180.591974:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079180.978352:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079181.424299:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079181.424309:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079181.424314:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079181.424324:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4863 00000100:00080000:1.0:1769079181.424334:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079181.424356:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079181.424386:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961568256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079181.424418:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079181.424422:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079181.424424:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079181.424429:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4863 00000100:00080000:1.0:1769079181.424433:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079181.424437:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079181.424445:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079181.424458:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961568384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079181.424466:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079181.424469:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079181.424470:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079181.424474:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4863 00000100:00080000:1.0:1769079181.424476:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079181.424479:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079181.424483:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079181.424491:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961568512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079181.424577:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961568256:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079181.424634:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079181.424684:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079181/real 1769079181] req@ffff895e4713d500 x1855008961568256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079181.424701:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961568256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079181.424710:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079181.424714:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079181.424717:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961568256:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079181.424724:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961568384:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079181.424788:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079181.424800:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079181/real 1769079181] req@ffff895e4713f480 x1855008961568384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079181.424812:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961568384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079181.424830:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079181.424834:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079181.424837:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961568384:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079181.424843:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961568512:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079181.424862:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079181.424873:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079181/real 1769079181] req@ffff895e4713ca80 x1855008961568512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079181.424884:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961568512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079181.424891:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079181.424894:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079181.424897:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961568512:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079181.618842:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079182.000092:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079182.640447:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079183.024241:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079183.664905:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079184.047966:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079184.687934:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079185.071863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079185.715393:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079186.096327:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079186.545373:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079186.545384:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079186.545388:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079186.545398:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4869 00000100:00080000:2.0:1769079186.545409:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079186.545415:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079186.545452:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961568640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079186.545482:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079186.545485:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079186.545486:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079186.545490:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4869 00000100:00080000:2.0:1769079186.545493:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079186.545497:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079186.545505:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079186.545517:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961568768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079186.545527:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079186.545530:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1769079186.545542:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961568640:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769079186.545547:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079186.545552:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4869 00000100:00080000:2.0:1769079186.545556:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079186.545560:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079186.545565:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079186.545576:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961568896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769079186.546112:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079186.546170:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079186/real 1769079186] req@ffff895e44e29f80 x1855008961568640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079186.546189:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961568640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079186.546198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079186.546202:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079186.546206:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961568640:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079186.546214:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961568768:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079186.546236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079186.546248:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079186/real 1769079186] req@ffff895e44e28e00 x1855008961568768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079186.546260:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961568768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079186.546268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079186.546271:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079186.546274:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961568768:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079186.546299:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961568896:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079186.546319:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079186.546330:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079186/real 1769079186] req@ffff895e44e2b480 x1855008961568896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079186.546341:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961568896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079186.546361:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079186.546365:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079186.546368:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961568896:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079186.735996:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079187.119982:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079187.760016:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079188.143866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079188.783990:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079189.167889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079189.810164:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079190.191889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079190.832437:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079191.215896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079191.663965:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079191.663977:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079191.663981:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079191.663992:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4874 00000100:00080000:1.0:1769079191.664003:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079191.664020:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079191.664057:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961569024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079191.664087:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079191.664090:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079191.664092:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079191.664096:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4874 00000100:00080000:1.0:1769079191.664100:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079191.664104:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079191.664112:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079191.664124:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961569152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079191.664133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1769079191.664140:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961569024:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769079191.664149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079191.664151:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079191.664155:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4874 00000100:00080000:1.0:1769079191.664158:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079191.664161:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079191.664165:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079191.664174:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961569280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769079191.664198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079191.664234:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079191/real 1769079191] req@ffff895e4713f800 x1855008961569024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079191.664251:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961569024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079191.664260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079191.664263:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079191.664267:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961569024:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079191.664289:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961569152:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079191.664311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079191.664324:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079191/real 1769079191] req@ffff895e4713dc00 x1855008961569152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079191.664336:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961569152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079191.664358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079191.664361:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079191.664364:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961569152:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079191.664370:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961569280:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079191.664391:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079191.664402:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079191/real 1769079191] req@ffff895e4713df80 x1855008961569280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079191.664412:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961569280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079191.664419:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079191.664422:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079191.664425:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961569280:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079191.856350:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079192.240064:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079192.880029:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079193.263848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079193.904047:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079194.287917:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079194.928000:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079195.312081:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079195.955405:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079196.335861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079196.783898:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079196.783906:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079196.783910:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079196.783917:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4879 00000100:00080000:3.0:1769079196.783937:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079196.783944:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079196.783981:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961569408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079196.784010:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079196.784015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079196.784017:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079196.784020:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4879 00000100:00080000:3.0:1769079196.784024:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079196.784028:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079196.784036:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079196.784050:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961569536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079196.784059:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079196.784061:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079196.784062:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079196.784064:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4879 00000100:00080000:3.0:1769079196.784067:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079196.784069:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079196.784084:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079196.784091:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961569664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079196.784324:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961569408:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079196.784665:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961569536:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079196.784686:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961569664:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079196.975958:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079197.359887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079198.001893:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079198.385132:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079199.028419:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079199.407868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769079199.856094:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769079199.856152:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769079199.856171:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769079199.865152:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079196/real 1769079199] req@ffff895e453d5500 x1855008961569408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079221 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079199.865193:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961569408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079221 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079199.865207:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079199.865212:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079199.865217:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961569408:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079199.865789:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769079199.865816:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769079199.865934:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079196/real 1769079199] req@ffff895e453d7b80 x1855008961569536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079199.865951:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961569536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079199.865959:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079199.865963:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079199.865966:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961569536:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079199.865972:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079196/real 1769079199] req@ffff895e453d7100 x1855008961569664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079199.865982:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961569664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079199.865990:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079199.865993:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079199.865996:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961569664:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079200.048526:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079200.431882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079201.074873:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079201.456982:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079201.903872:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079201.903884:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079201.903887:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079201.903906:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4884 00000100:00080000:3.0:1769079201.903917:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079201.903923:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079201.909906:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961569792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079201.909940:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079201.909945:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079201.909948:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079201.909954:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4884 00000100:00080000:3.0:1769079201.909966:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1769079201.909968:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961569792:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769079201.909976:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079201.909988:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079201.910007:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961569920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079201.910016:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079201.910020:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079201.910022:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079201.910026:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4884 00000100:00080000:3.0:1769079201.910048:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079201.910055:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079201.910060:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1769079201.910064:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079201.910074:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961570048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079201.910115:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079201/real 1769079201] req@ffff895e453d4700 x1855008961569792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079201.910134:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961569792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079201.910144:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079201.910148:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079201.910157:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961569792:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079201.910179:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961569920:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079201.910201:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079201.910211:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079201/real 1769079201] req@ffff895e453d5880 x1855008961569920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079201.910222:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961569920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079201.910230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079201.910233:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079201.910236:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961569920:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079201.910240:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961570048:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079201.910253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079201.910261:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079201/real 1769079201] req@ffff895e453d5180 x1855008961570048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079201.910271:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961570048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079201.910278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079201.910280:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079201.910283:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961570048:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079202.097491:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079202.480198:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079203.120922:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079203.503847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079204.147497:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079204.529263:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079205.168826:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079205.552017:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079206.197519:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079206.576162:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079207.024049:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079207.024065:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079207.024071:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079207.024086:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4889 00000100:00080000:2.0:1769079207.024104:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079207.024115:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079207.024165:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961570176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079207.024224:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079207.024228:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079207.024230:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079207.024236:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4889 00000100:00080000:2.0:1769079207.024240:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079207.024244:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079207.024254:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079207.024267:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961570304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079207.024276:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079207.024279:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079207.024281:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079207.024285:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4889 00000100:00080000:2.0:1769079207.024289:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079207.024294:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079207.024298:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079207.024309:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961570432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079207.025443:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961570176:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079207.025541:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079207.025595:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079207/real 1769079207] req@ffff895e44e2b100 x1855008961570176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079207.025615:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961570176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079207.025626:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079207.025631:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079207.025635:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961570176:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079207.025643:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961570304:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079207.025667:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079207.025679:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079207/real 1769079207] req@ffff895e44e2a300 x1855008961570304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079207.025689:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961570304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079207.025698:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079207.025701:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079207.025704:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961570304:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079207.025711:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961570432:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079207.025729:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079207.025801:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079207/real 1769079207] req@ffff895e44e2aa00 x1855008961570432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079207.025812:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961570432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079207.025821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079207.025824:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079207.025827:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961570432:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079207.220050:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079207.600884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079208.240115:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079208.623926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079209.264251:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079209.649479:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079210.289070:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079210.672376:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079211.312827:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079211.699283:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079212.145012:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079212.145030:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079212.145036:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079212.145049:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4894 00000100:00080000:3.0:1769079212.145065:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079212.145075:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079212.145129:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961570560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079212.145251:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961570560:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079212.145344:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769079212.145391:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079212.145395:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079212.145397:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079212.145402:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4894 00000100:00080000:3.0:1769079212.145406:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079212.145410:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079212.145422:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079212.145436:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961570688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079212.148038:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079212/real 1769079212] req@ffff895e453d6300 x1855008961570560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079212.148079:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961570560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079212.148093:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079212.148098:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079212.148103:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961570560:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079212.148149:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961570688:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079212.148231:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079212.148284:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079212/real 1769079212] req@ffff895e453d4380 x1855008961570688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079212.148298:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961570688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079212.148307:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079212.148311:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079212.148315:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961570688:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769079212.149149:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079212.149170:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079212.149175:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079212.149184:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4894 00000100:00080000:3.0:1769079212.149194:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079212.149202:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079212.149217:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079212.149261:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961570816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079212.149436:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961570816:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079212.149577:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079212.149604:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079212/real 1769079212] req@ffff895e453d6680 x1855008961570816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079212.149626:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961570816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079212.149634:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079212.149638:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079212.149641:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961570816:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079212.336112:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079212.721606:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079213.360971:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079213.745553:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079214.389077:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079214.773723:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079215.407958:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079215.791859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079216.431954:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079216.817645:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079217.266395:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079217.266406:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079217.266411:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079217.266420:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4899 00000100:00080000:3.0:1769079217.266441:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079217.266449:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079217.266486:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961570944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079217.266522:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079217.266526:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079217.266527:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079217.266531:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4899 00000100:00080000:3.0:1769079217.266535:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079217.266538:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079217.266562:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079217.266576:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961571072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079217.266585:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079217.266588:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079217.266590:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079217.266593:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4899 00000100:00080000:3.0:1769079217.266597:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079217.266601:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079217.266605:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079217.266616:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961571200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079217.266688:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961570944:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079217.266838:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079217.266892:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079217/real 1769079217] req@ffff895e453d6d80 x1855008961570944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079217.266933:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961570944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079217.266944:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079217.266950:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079217.266954:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961570944:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079217.266963:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961571072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079217.266988:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079217.267001:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079217/real 1769079217] req@ffff895e453d6680 x1855008961571072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079217.267013:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961571072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079217.267034:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079217.267038:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079217.267042:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961571072:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079217.267048:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961571200:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079217.267069:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079217.267080:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079217/real 1769079217] req@ffff895e453d4380 x1855008961571200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079217.267092:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961571200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079217.267113:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079217.267117:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079217.267120:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961571200:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079217.465077:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079217.839865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079218.480465:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079218.863969:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079219.503972:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079219.887859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079220.528137:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079220.917490:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079221.551962:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079221.935858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079222.397352:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079222.397368:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079222.397373:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079222.397383:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4904 00000100:00080000:1.0:1769079222.397394:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079222.397400:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079222.397437:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961571328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079222.397466:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079222.397469:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079222.397471:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079222.397476:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4904 00000100:00080000:1.0:1769079222.397479:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079222.397483:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079222.397491:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079222.397502:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961571456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079222.397512:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079222.397514:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079222.397533:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079222.397537:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4904 00000100:00080000:1.0:1769079222.397541:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079222.397545:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079222.397550:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079222.397561:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961571584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079222.397734:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961571328:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079222.397837:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079222.397874:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079222/real 1769079222] req@ffff895e4713d180 x1855008961571328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079222.397891:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961571328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079222.397900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079222.397904:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079222.397918:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961571328:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079222.397925:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961571456:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079222.397946:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079222.397958:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079222/real 1769079222] req@ffff895e4713c380 x1855008961571456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079222.397983:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961571456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079222.397991:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079222.397995:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079222.397998:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961571456:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079222.398003:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961571584:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079222.398024:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079222.398036:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079222/real 1769079222] req@ffff895e4713df80 x1855008961571584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079222.398048:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961571584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079222.398055:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079222.398058:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079222.398060:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961571584:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079222.576512:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079222.961006:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079223.600299:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079223.985021:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079224.624519:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079225.009011:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079225.647976:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079226.033667:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079226.676041:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079227.055862:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079227.503879:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079227.503887:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079227.503890:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079227.503899:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4910 00000100:00080000:2.0:1769079227.503908:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079227.503914:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079227.503946:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961571712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079227.503970:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079227.503972:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079227.503973:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079227.503976:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4910 00000100:00080000:2.0:1769079227.503986:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079227.503990:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079227.503999:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079227.504014:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961571840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079227.504022:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079227.504025:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079227.504026:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079227.504030:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4910 00000100:00080000:2.0:1769079227.504033:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079227.504036:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079227.504056:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079227.504068:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961571968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079227.504165:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961571712:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079227.504223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079227.504258:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079227/real 1769079227] req@ffff895e44e28000 x1855008961571712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079227.504287:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961571712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079227.504297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079227.504301:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079227.504305:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961571712:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079227.504312:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961571840:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079227.504334:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079227.504345:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079227/real 1769079227] req@ffff895e44e28380 x1855008961571840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079227.504356:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961571840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079227.504366:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079227.504368:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079227.504371:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961571840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079227.504376:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961571968:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079227.504393:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079227.504402:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079227/real 1769079227] req@ffff895e44e29880 x1855008961571968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079227.504411:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961571968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079227.504448:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079227.504451:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079227.504454:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961571968:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079227.697282:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079228.085685:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079228.720081:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079229.103866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079229.746709:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079230.128442:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079230.768029:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079231.154283:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079231.792339:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079232.176456:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079232.623935:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079232.623954:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079232.623976:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079232.623987:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4915 00000100:00080000:3.0:1769079232.623999:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079232.624007:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079232.624047:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961572096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079232.624077:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079232.624080:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079232.624082:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079232.624086:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4915 00000100:00080000:3.0:1769079232.624090:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079232.624094:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079232.624103:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079232.624114:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961572224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079232.624122:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079232.624125:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079232.624126:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079232.624130:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4915 00000100:00080000:3.0:1769079232.624133:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079232.624136:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079232.624141:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079232.624149:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961572352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079232.624407:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961572096:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079232.624468:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079232.624503:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079232/real 1769079232] req@ffff895e453d6300 x1855008961572096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079232.624519:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961572096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079232.624527:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079232.624530:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079232.624533:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961572096:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079232.624540:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961572224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079232.624561:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079232.624574:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079232/real 1769079232] req@ffff895e453d5180 x1855008961572224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079232.624585:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961572224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079232.624606:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079232.624611:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079232.624614:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961572224:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079232.624620:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961572352:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079232.624644:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079232.624655:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079232/real 1769079232] req@ffff895e453d5880 x1855008961572352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079232.624667:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961572352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079232.624675:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079232.624678:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079232.624681:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961572352:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079232.815938:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079233.199898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079233.840046:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079234.223966:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079234.865849:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079235.247921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079235.893011:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079236.271904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079236.912647:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079237.297142:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079237.743874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079237.743885:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079237.743889:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079237.743900:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4920 00000100:00080000:3.0:1769079237.743910:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079237.743916:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079237.743951:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961572480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079237.743993:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079237.743997:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079237.743998:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079237.744003:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4920 00000100:00080000:3.0:1769079237.744013:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079237.744018:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079237.744027:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079237.744038:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961572608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079237.744047:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079237.744051:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079237.744052:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079237.744056:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4920 00000100:00080000:3.0:1769079237.744059:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079237.744062:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079237.744081:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079237.744093:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961572736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079237.744255:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961572480:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079237.744316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079237.744353:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079237/real 1769079237] req@ffff895e453d4700 x1855008961572480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079237.744370:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961572480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079237.744393:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079237.744397:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079237.744401:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961572480:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079237.744408:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961572608:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079237.744432:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079237.744444:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079237/real 1769079237] req@ffff895e453d7100 x1855008961572608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079237.744455:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961572608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079237.744463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079237.744466:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079237.744468:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961572608:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079237.744474:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961572736:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079237.744493:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079237.744503:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079237/real 1769079237] req@ffff895e453d7b80 x1855008961572736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079237.744513:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961572736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079237.744519:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079237.744522:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079237.744524:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961572736:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079237.936239:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079238.319880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079238.963040:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079239.344524:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079239.985020:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079240.367907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079241.007986:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079241.392024:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079242.031984:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079242.416068:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079242.868532:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079242.868542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079242.868546:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079242.868557:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4925 00000100:00080000:3.0:1769079242.868568:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079242.868575:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079242.868612:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961572864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079242.868646:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079242.868650:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079242.868652:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079242.868656:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4925 00000100:00080000:3.0:1769079242.868660:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079242.868664:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079242.868673:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079242.868688:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961572992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079242.868697:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079242.868700:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079242.868701:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079242.868705:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4925 00000100:00080000:3.0:1769079242.868735:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079242.868740:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079242.868774:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079242.868787:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961573120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079242.873377:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961572864:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079242.873456:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079242.873500:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079242/real 1769079242] req@ffff895e453d5500 x1855008961572864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079242.873518:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961572864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079242.873529:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079242.873534:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079242.873538:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961572864:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079242.873546:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961572992:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079242.873569:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079242.873581:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079242/real 1769079242] req@ffff895e453d4e00 x1855008961572992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079242.873594:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961572992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079242.873603:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079242.873606:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079242.873610:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961572992:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079242.873634:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961573120:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079242.873657:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079242.873669:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079242/real 1769079242] req@ffff895e453d4a80 x1855008961573120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079242.873682:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961573120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079242.873690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079242.873694:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079242.873697:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961573120:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079243.056944:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079243.442133:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079244.079985:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079244.466028:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079245.103958:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079245.487906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079246.134591:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079246.511888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079247.151950:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079247.538162:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079247.987737:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079247.987763:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079247.987768:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079247.987778:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4930 00000100:00080000:0.0:1769079247.987790:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079247.987797:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079247.987833:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961573248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079247.987858:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079247.987862:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079247.987864:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079247.987883:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4930 00000100:00080000:0.0:1769079247.987886:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079247.987890:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079247.987900:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079247.987912:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961573376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079247.987921:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079247.987924:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079247.987926:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079247.987942:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4930 00000100:00080000:0.0:1769079247.987946:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079247.987949:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079247.987954:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079247.987965:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961573504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079247.988171:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961573248:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079247.988235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079247.988274:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079247/real 1769079247] req@ffff895e4383c700 x1855008961573248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079247.988321:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961573248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079247.988332:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079247.988336:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079247.988341:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961573248:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079247.988348:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961573376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079247.988386:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079247.988400:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079247/real 1769079247] req@ffff895e4383e680 x1855008961573376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079247.988412:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961573376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079247.988420:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079247.988423:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079247.988426:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961573376:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079247.988433:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961573504:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079247.988452:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079247.988464:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079247/real 1769079247] req@ffff895e4383d880 x1855008961573504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079247.988475:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961573504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079247.988494:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079247.988498:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079247.988501:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961573504:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079248.176010:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079248.559879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079249.199942:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079249.583886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079250.226703:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079250.607935:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079251.247998:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079251.631869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079252.277110:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079252.655957:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079253.104824:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079253.104833:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079253.104837:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079253.104847:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4935 00000100:00080000:0.0:1769079253.104858:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079253.104865:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079253.104915:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961573632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079253.104945:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079253.104948:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079253.104950:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079253.104953:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4935 00000100:00080000:0.0:1769079253.104957:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079253.104961:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079253.104969:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079253.104978:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961573760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079253.104987:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079253.104990:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079253.104991:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079253.104994:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4935 00000100:00080000:0.0:1769079253.104996:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079253.105000:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079253.105003:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079253.105012:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961573888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079253.105036:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961573632:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079253.105104:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079253.105157:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079253/real 1769079253] req@ffff895e4383ed80 x1855008961573632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079253.105175:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961573632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079253.105185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079253.105190:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079253.105193:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961573632:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079253.105201:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961573760:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079253.105223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079253.105236:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079253/real 1769079253] req@ffff895e4383df80 x1855008961573760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079253.105248:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961573760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079253.105263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079253.105267:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079253.105270:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961573760:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079253.105276:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961573888:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079253.105300:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079253.105311:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079253/real 1769079253] req@ffff895e4383d180 x1855008961573888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079253.105322:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961573888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079253.105341:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079253.105345:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079253.105607:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961573888:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079253.297045:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079253.679886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079254.320053:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079254.703868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079255.343968:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079255.727871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079256.368158:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079256.752023:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079257.392050:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079257.775902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079258.225332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079258.225345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079258.225349:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079258.225359:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4940 00000100:00080000:2.0:1769079258.225370:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079258.225377:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079258.225419:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961574016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079258.225453:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079258.225457:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079258.225459:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079258.225472:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4940 00000100:00080000:2.0:1769079258.225477:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079258.225496:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079258.225506:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079258.225519:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961574144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079258.225529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079258.225533:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079258.225535:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079258.225538:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4940 00000100:00080000:2.0:1769079258.225543:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079258.225546:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079258.225550:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079258.225559:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961574272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079258.225654:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961574016:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079258.225715:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079258.225796:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079258/real 1769079258] req@ffff895e44e29880 x1855008961574016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079258.225816:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961574016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079258.225831:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079258.225836:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079258.225840:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961574016:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079258.225848:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961574144:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079258.225872:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079258.225896:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079258/real 1769079258] req@ffff895e44e28380 x1855008961574144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079258.225908:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961574144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079258.225915:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079258.225918:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079258.225921:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961574144:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079258.225927:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961574272:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079258.225946:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079258.225957:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079258/real 1769079258] req@ffff895e44e28000 x1855008961574272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079258.225967:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961574272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079258.225974:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079258.225976:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079258.225979:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961574272:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079258.416517:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079258.800469:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079259.440409:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079259.823884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079260.465469:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079260.847881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079261.488303:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079261.871973:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079262.512052:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079262.895898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079263.348517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079263.348529:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079263.348534:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079263.348544:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4945 00000100:00080000:3.0:1769079263.348556:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079263.348563:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079263.348612:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961574400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079263.348644:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079263.348647:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079263.348649:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079263.348653:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4945 00000100:00080000:3.0:1769079263.348657:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079263.348661:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079263.348669:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079263.348682:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961574528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079263.348691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079263.348710:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079263.348712:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079263.348716:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4945 00000100:00080000:3.0:1769079263.348720:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079263.348723:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079263.348728:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079263.348738:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961574656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079263.351703:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961574400:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079263.351867:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961574528:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079263.351897:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961574656:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079263.536302:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079263.919886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079264.559995:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079264.944096:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079265.584058:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079265.967867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769079266.416086:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769079266.416142:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769079266.416161:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769079266.416202:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769079266.416212:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:1.0:1769079266.416332:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079263/real 1769079266] req@ffff895e453d7480 x1855008961574400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079266.416357:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961574400/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079266.416368:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079266.416373:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079266.416377:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961574400:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079266.416385:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079263/real 1769079266] req@ffff895e453d5f80 x1855008961574528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079266.416397:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961574528/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079266.416405:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079266.416422:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079266.416426:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961574528:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079266.416432:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079263/real 1769079266] req@ffff895e453d5c00 x1855008961574656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079266.416446:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961574656/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079266.416453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079266.416457:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079266.416460:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961574656:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079266.608055:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079266.991897:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079267.634145:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079268.015889:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079268.469384:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079268.469395:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079268.469400:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079268.469421:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4951 00000100:00080000:1.0:1769079268.469436:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079268.469444:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079268.469483:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961574784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079268.469515:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079268.469519:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079268.469520:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079268.469524:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4951 00000100:00080000:1.0:1769079268.469545:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079268.469549:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079268.469559:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079268.469573:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961574912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079268.469583:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079268.469586:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079268.469587:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079268.469591:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4951 00000100:00080000:1.0:1769079268.469593:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079268.469597:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079268.469601:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079268.469610:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961575040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079268.469698:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961574784:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079268.469796:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079268.469844:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079268/real 1769079268] req@ffff895e4713dc00 x1855008961574784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079268.469863:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961574784/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079268.469872:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079268.469876:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079268.469879:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961574784:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079268.469887:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961574912:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079268.469911:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079268.469924:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079268/real 1769079268] req@ffff895e4713f800 x1855008961574912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079268.469934:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961574912/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079268.469941:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079268.469944:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079268.469947:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961574912:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079268.469964:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961575040:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079268.469987:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079268.469998:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079268/real 1769079268] req@ffff895e4713ca80 x1855008961575040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079268.470008:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961575040/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079268.470015:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079268.470018:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079268.470021:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961575040:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079268.655980:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079269.039877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079269.680060:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079270.064829:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079270.704159:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079271.088044:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079271.728558:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079272.111891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079272.757822:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079273.136432:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079273.583880:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079273.583891:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079273.583895:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079273.583904:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4956 00000100:00080000:1.0:1769079273.583915:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079273.583922:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079273.583974:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961575168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079273.584006:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079273.584010:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079273.584012:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079273.584016:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4956 00000100:00080000:1.0:1769079273.584020:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079273.584024:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079273.584033:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079273.584044:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961575296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079273.584053:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079273.584056:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079273.584057:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079273.584060:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4956 00000100:00080000:1.0:1769079273.584063:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079273.584066:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079273.584071:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079273.584078:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961575424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079273.586008:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961575168:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079273.586081:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079273.586125:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079273/real 1769079273] req@ffff895e4713f480 x1855008961575168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079273.586144:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961575168/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079273.586154:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079273.586158:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079273.586162:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961575168:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079273.586184:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961575296:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079273.586222:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079273.586235:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079273/real 1769079273] req@ffff895e4713d500 x1855008961575296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079273.586247:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961575296/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079273.586254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079273.586257:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079273.586260:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961575296:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079273.586265:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961575424:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079273.586284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079273.586295:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079273/real 1769079273] req@ffff895e4713ce00 x1855008961575424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079273.586306:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961575424/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079273.586313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079273.586316:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079273.586319:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961575424:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079273.776078:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079274.159892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079274.800009:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079275.184004:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079275.823996:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079276.208552:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079276.849615:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079277.231879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079277.871993:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079278.255981:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079278.703944:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079278.703955:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079278.703959:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079278.703969:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4961 00000100:00080000:3.0:1769079278.703980:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079278.703988:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079278.704024:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961575552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079278.704056:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079278.704060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079278.704062:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079278.704066:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4961 00000100:00080000:3.0:1769079278.704070:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079278.704073:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079278.704081:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079278.704094:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961575680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079278.704132:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079278.704135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079278.704137:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079278.704141:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4961 00000100:00080000:3.0:1769079278.704145:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079278.704149:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079278.704154:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079278.704166:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961575808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079278.704272:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961575552:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079278.704341:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079278.704377:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079278/real 1769079278] req@ffff895e453d7800 x1855008961575552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079278.704395:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961575552/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079278.704403:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079278.704407:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079278.704410:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961575552:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079278.704417:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961575680:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079278.704439:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079278.704450:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079278/real 1769079278] req@ffff895e453d5c00 x1855008961575680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079278.704460:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961575680/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079278.704468:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079278.704471:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079278.704474:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961575680:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079278.704480:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961575808:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079278.704512:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079278.704523:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079278/real 1769079278] req@ffff895e453d5f80 x1855008961575808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079278.704547:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961575808/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079278.704555:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079278.704558:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079278.704560:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961575808:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079278.896045:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079279.279920:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079279.919982:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079280.303856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079280.945683:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079281.327870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079281.968331:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079282.351890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079282.992545:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079283.375865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079283.823992:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079283.824004:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079283.824007:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079283.824016:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4966 00000100:00080000:1.0:1769079283.824045:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079283.824054:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079283.824092:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961575936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079283.824125:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079283.824128:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079283.824130:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079283.824134:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4966 00000100:00080000:1.0:1769079283.824138:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079283.824142:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079283.824152:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079283.824163:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961576064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079283.824172:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079283.824175:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079283.824195:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079283.824199:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4966 00000100:00080000:1.0:1769079283.824203:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079283.824208:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079283.824213:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079283.824224:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961576192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079283.824398:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961575936:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079283.824459:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079283.824501:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079283/real 1769079283] req@ffff895e4713ea00 x1855008961575936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079283.824517:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961575936/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079283.824526:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079283.824530:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079283.824534:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961575936:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079283.824540:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961576064:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079283.824559:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079283.824571:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079283/real 1769079283] req@ffff895e4713f100 x1855008961576064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079283.824597:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961576064/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079283.824605:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079283.824609:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079283.824612:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961576064:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079283.824618:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961576192:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079283.824640:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079283.824651:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079283/real 1769079283] req@ffff895e4713e300 x1855008961576192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079283.824660:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961576192/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079283.824666:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079283.824669:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079283.824672:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961576192:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079284.016781:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079284.399831:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079285.040416:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079285.423891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079286.063986:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079286.447916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079287.091368:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079287.472527:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079288.112039:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079288.495916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079288.950330:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079288.950341:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079288.950346:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079288.950356:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4971 00000100:00080000:0.0:1769079288.950385:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079288.950392:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079288.950427:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961576320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079288.950507:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961576320:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079288.950580:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769079288.950617:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079288.950622:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079288.950624:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079288.950628:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4971 00000100:00080000:0.0:1769079288.950632:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079288.950636:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079288.950646:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079288.950659:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961576448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079288.953989:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079288/real 1769079288] req@ffff895e4383d180 x1855008961576320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079288.954195:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961576320/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079288.954208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079288.954213:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079288.954217:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961576320:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079288.954250:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961576448:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079288.954311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079288.954327:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079288/real 1769079288] req@ffff895e4383df80 x1855008961576448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079288.954355:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961576448/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079288.954364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079288.954367:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079288.954371:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961576448:192.168.204.117@tcp:38:kworker.0 00000100:00080000:0.0:1769079288.954490:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079288.954495:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079288.954498:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079288.954502:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4971 00000100:00080000:0.0:1769079288.954509:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079288.954515:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079288.954525:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079288.954715:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961576576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079288.954800:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961576576:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079288.954824:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079288.954836:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079288/real 1769079288] req@ffff895e4383ed80 x1855008961576576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079288.954849:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961576576/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079288.954857:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079288.954861:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079288.954864:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961576576:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079289.139804:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079289.520228:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079290.164823:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079290.548056:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079291.185121:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079291.567918:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079292.208029:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079292.593114:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079293.233841:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079293.615883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079294.064054:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079294.064065:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079294.064070:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079294.064079:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4976 00000100:00080000:3.0:1769079294.064090:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079294.064098:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079294.064136:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961576704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079294.064162:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079294.064165:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079294.064166:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079294.064169:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4976 00000100:00080000:3.0:1769079294.064172:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079294.064175:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079294.064203:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079294.064218:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961576832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079294.064228:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079294.064230:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079294.064232:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079294.064245:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4976 00000100:00080000:3.0:1769079294.064249:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079294.064253:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079294.064258:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079294.064267:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961576960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079294.064344:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961576704:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079294.064402:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079294.064430:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079294/real 1769079294] req@ffff895e453d7480 x1855008961576704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079294.064448:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961576704/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079294.064455:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079294.064458:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079294.064460:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961576704:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079294.064465:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961576832:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079294.064479:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079294.064486:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079294/real 1769079294] req@ffff895e453d4a80 x1855008961576832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079294.064493:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961576832/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079294.064498:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079294.064500:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079294.064501:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961576832:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079294.064504:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961576960:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079294.064522:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079294.064529:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079294/real 1769079294] req@ffff895e453d4e00 x1855008961576960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079294.064536:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961576960/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079294.064548:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079294.064550:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079294.064552:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961576960:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079294.256038:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079294.640177:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079295.280876:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079295.663849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079296.304119:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079296.690723:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079297.328363:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079297.711872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079298.352084:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079298.735924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079299.183940:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079299.183951:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079299.183955:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079299.183966:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4981 00000100:00080000:1.0:1769079299.183978:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079299.183985:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079299.184023:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961577088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079299.184057:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079299.184061:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079299.184063:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079299.184067:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4981 00000100:00080000:1.0:1769079299.184070:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079299.184073:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079299.184083:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079299.184096:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961577216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079299.184121:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079299.184124:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079299.184126:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079299.184129:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4981 00000100:00080000:1.0:1769079299.184134:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079299.184137:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079299.184154:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079299.184166:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961577344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079299.184274:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961577088:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079299.184342:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079299.184381:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079299/real 1769079299] req@ffff895e4713d880 x1855008961577088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079299.184398:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961577088/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079299.184408:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079299.184412:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079299.184415:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961577088:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079299.184422:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961577216:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079299.184442:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079299.184455:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079299/real 1769079299] req@ffff895e4713ed80 x1855008961577216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079299.184466:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961577216/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079299.184474:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079299.184477:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079299.184480:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961577216:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079299.184499:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961577344:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079299.184530:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079299.184542:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079299/real 1769079299] req@ffff895e4713c000 x1855008961577344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079299.184552:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961577344/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079299.184558:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079299.184561:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079299.184564:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961577344:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079299.376582:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079299.759941:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079300.400238:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079300.783944:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079301.423956:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079301.807824:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079302.447976:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079302.833244:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079303.472064:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079303.855894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079304.306596:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079304.306606:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079304.306611:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079304.306620:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4986 00000100:00080000:3.0:1769079304.306631:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079304.306638:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079304.306676:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961577472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079304.306716:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079304.306719:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079304.306728:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079304.306732:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4986 00000100:00080000:3.0:1769079304.306782:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079304.306788:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079304.306798:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079304.306809:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961577600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079304.306818:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079304.306821:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079304.306823:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079304.306826:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4986 00000100:00080000:3.0:1769079304.306829:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079304.306832:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079304.306836:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079304.306845:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961577728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079304.307156:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961577472:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079304.307214:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079304.307246:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079304/real 1769079304] req@ffff895e453d5500 x1855008961577472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079304.307262:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961577472/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079304.307271:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079304.307274:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079304.307278:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961577472:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079304.307293:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961577600:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079304.307316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079304.307328:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079304/real 1769079304] req@ffff895e453d7b80 x1855008961577600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079304.307339:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961577600/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079304.307346:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079304.307349:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079304.307352:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961577600:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079304.307357:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961577728:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079304.307374:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079304.307384:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079304/real 1769079304] req@ffff895e453d7100 x1855008961577728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079304.307395:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961577728/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079304.307403:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079304.307406:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079304.307409:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961577728:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079304.496924:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079304.879867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079305.519964:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079305.903887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079306.544152:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079306.927885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079307.567986:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079307.951874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079308.592672:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079308.976096:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079309.424066:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079309.424077:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079309.424082:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079309.424093:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4991 00000100:00080000:3.0:1769079309.424106:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079309.424114:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079309.424153:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961577856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079309.424182:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079309.424200:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079309.424202:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079309.424206:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4991 00000100:00080000:3.0:1769079309.424210:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079309.424214:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079309.424222:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079309.424234:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961577984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079309.424243:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079309.424245:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079309.424247:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079309.424250:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4991 00000100:00080000:3.0:1769079309.424253:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079309.424257:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079309.424260:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079309.424269:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961578112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079309.424395:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961577856:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079309.424463:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079309.424501:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079309/real 1769079309] req@ffff895e453d4700 x1855008961577856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079309.424519:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961577856/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079309.424530:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079309.424534:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079309.424538:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961577856:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079309.424545:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961577984:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079309.424566:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079309.424578:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079309/real 1769079309] req@ffff895e453d5880 x1855008961577984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079309.424589:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961577984/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079309.424596:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079309.424627:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079309.424630:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961577984:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079309.424636:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961578112:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079309.424655:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079309.424665:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079309/real 1769079309] req@ffff895e453d5180 x1855008961578112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079309.424689:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961578112/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079309.424696:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079309.424699:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079309.424701:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961578112:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079309.618385:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079309.999883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079310.639981:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079311.025097:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079311.664046:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079312.047885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079312.687963:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079313.071825:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079313.712073:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079314.095879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079314.543958:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079314.543970:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079314.543976:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079314.543986:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 4997 00000100:00080000:3.0:1769079314.543996:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079314.544003:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079314.544041:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961578240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079314.544088:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079314.544092:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079314.544094:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079314.544098:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4997 00000100:00080000:3.0:1769079314.544102:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079314.544107:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079314.544116:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079314.544131:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961578368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079314.544140:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079314.544143:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079314.544145:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079314.544148:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 4997 00000100:00080000:3.0:1769079314.544152:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079314.544156:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079314.544160:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079314.544169:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961578496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079314.544343:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961578240:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079314.544402:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079314.544440:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079314/real 1769079314] req@ffff895e453d6300 x1855008961578240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079314.544456:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961578240/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079314.544465:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079314.544469:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079314.544472:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961578240:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079314.544479:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961578368:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079314.544499:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079314.544511:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079314/real 1769079314] req@ffff895e453d4380 x1855008961578368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079314.544547:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961578368/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079314.544554:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079314.544557:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079314.544560:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961578368:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079314.544565:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961578496:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079314.544586:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079314.544597:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079314/real 1769079314] req@ffff895e453d6680 x1855008961578496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079314.544608:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961578496/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079314.544615:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079314.544618:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079314.544621:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961578496:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079314.736029:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079315.119870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079315.762660:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079316.143958:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079316.784633:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079317.168245:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079317.809552:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079318.193671:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079318.832034:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079319.215888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079319.664015:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079319.664026:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079319.664030:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079319.664039:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5002 00000100:00080000:1.0:1769079319.664051:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079319.664058:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079319.664094:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961578624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079319.664126:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079319.664129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079319.664131:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079319.664135:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5002 00000100:00080000:1.0:1769079319.664138:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079319.664142:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079319.664150:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079319.664161:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961578752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079319.664170:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079319.664172:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079319.664174:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079319.664177:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5002 00000100:00080000:1.0:1769079319.664181:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079319.664184:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079319.664204:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079319.664216:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961578880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079319.664331:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961578624:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079319.664396:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079319.664460:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079319/real 1769079319] req@ffff895e4713fb80 x1855008961578624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079319.664491:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961578624/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079319.664502:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079319.664506:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079319.664509:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961578624:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079319.664517:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961578752:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079319.664539:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079319.664551:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079319/real 1769079319] req@ffff895e4713c700 x1855008961578752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079319.664562:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961578752/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079319.664570:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079319.664573:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079319.664576:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961578752:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079319.664582:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961578880:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079319.664600:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079319.664611:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079319/real 1769079319] req@ffff895e4713c000 x1855008961578880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079319.664623:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961578880/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079319.664643:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079319.664646:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079319.664650:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961578880:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079319.855987:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079320.239930:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079320.880162:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079321.263877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079321.908169:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079322.288003:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079322.928792:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079323.311987:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079323.960047:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079324.335895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079324.784790:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079324.784800:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079324.784804:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079324.784813:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5007 00000100:00080000:3.0:1769079324.784840:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079324.784857:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079324.784890:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961579008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079324.784933:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079324.784937:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079324.784939:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079324.784943:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5007 00000100:00080000:3.0:1769079324.784946:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079324.784950:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079324.784959:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079324.784971:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961579136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079324.784980:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079324.784983:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079324.784985:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079324.784988:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5007 00000100:00080000:3.0:1769079324.784992:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079324.784995:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079324.784999:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079324.785009:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961579264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079324.785090:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961579008:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079324.785172:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079324.785221:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079324/real 1769079324] req@ffff895e453d6d80 x1855008961579008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079324.785239:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961579008/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079324.785249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079324.785253:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079324.785256:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961579008:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079324.785263:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961579136:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079324.785284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079324.785296:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079324/real 1769079324] req@ffff895e453d6680 x1855008961579136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079324.785307:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961579136/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079324.785328:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079324.785331:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079324.785334:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961579136:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079324.785350:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961579264:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079324.785372:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079324.785382:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079324/real 1769079324] req@ffff895e453d4380 x1855008961579264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079324.785404:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961579264/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079324.785411:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079324.785414:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079324.785417:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961579264:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079324.977072:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079325.360042:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079326.001441:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079326.384004:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079327.026040:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079327.407905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079328.048441:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079328.431888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079329.071968:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079329.455863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079329.903894:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079329.903919:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079329.903923:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079329.903932:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5012 00000100:00080000:3.0:1769079329.903943:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079329.903950:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079329.903982:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961579392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079329.904015:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079329.904017:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079329.904019:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079329.904023:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5012 00000100:00080000:3.0:1769079329.904027:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079329.904030:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079329.904038:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079329.904048:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961579520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079329.904056:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079329.904059:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079329.904061:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079329.904065:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5012 00000100:00080000:3.0:1769079329.904069:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079329.904072:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079329.904077:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079329.904086:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961579648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079329.904093:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961579392:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079329.904200:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961579520:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079329.904237:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961579648:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079330.095940:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079330.479861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079331.120003:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079331.503930:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079332.145362:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079332.528360:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:2.0:1769079332.976262:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:2.0:1769079332.976326:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1769079332.976345:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769079332.976393:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:2.0:1769079332.976402:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769079332.976639:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079329/real 1769079332] req@ffff895e453d5880 x1855008961579648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079332.976683:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961579648/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079332.976696:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079332.976701:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079332.976706:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961579648:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079332.976713:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079329/real 1769079332] req@ffff895e453d6300 x1855008961579392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079332.976725:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961579392/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079332.976732:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079332.976735:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079332.976738:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961579392:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079332.976773:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079329/real 1769079332] req@ffff895e453d5180 x1855008961579520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079332.976790:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961579520/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079332.976797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079332.976801:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079332.976804:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961579520:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079333.168006:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079333.551906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079334.191976:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079334.576208:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079335.023868:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079335.023879:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079335.023883:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079335.023892:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5017 00000100:00080000:3.0:1769079335.023903:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079335.023911:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079335.023948:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961579776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079335.023987:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079335.023991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079335.023993:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079335.023997:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5017 00000100:00080000:3.0:1769079335.024000:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079335.024004:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079335.024013:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079335.024026:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961579904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079335.024036:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079335.024039:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079335.024041:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079335.024044:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5017 00000100:00080000:3.0:1769079335.024047:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079335.024051:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079335.024071:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079335.024081:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961580032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079335.024922:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961579776:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079335.024990:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079335.025029:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079335/real 1769079335] req@ffff895e453d4700 x1855008961579776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079335.025048:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961579776/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079335.025058:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079335.025062:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079335.025066:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961579776:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079335.025074:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961579904:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079335.025093:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079335.025108:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079335/real 1769079335] req@ffff895e453d7100 x1855008961579904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079335.025118:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961579904/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079335.025124:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079335.025128:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079335.025130:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961579904:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079335.025137:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961580032:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079335.025157:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079335.025166:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079335/real 1769079335] req@ffff895e453d7b80 x1855008961580032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079335.025200:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961580032/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079335.025208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079335.025212:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079335.025215:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961580032:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079335.216284:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079335.599966:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079336.240317:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079336.623876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079337.263967:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079337.648211:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079338.291377:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079338.672083:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079339.312072:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079339.695910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079340.144152:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079340.144166:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079340.144172:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079340.144185:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5022 00000100:00080000:1.0:1769079340.144198:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079340.144206:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079340.144276:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961580160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079340.144315:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079340.144319:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079340.144321:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079340.144325:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5022 00000100:00080000:1.0:1769079340.144329:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079340.144333:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079340.144343:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079340.144353:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961580288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079340.144363:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079340.144366:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079340.144367:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079340.144371:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5022 00000100:00080000:1.0:1769079340.144375:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079340.144379:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079340.144383:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079340.144392:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961580416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079340.147406:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961580160:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079340.147484:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079340.147526:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079340/real 1769079340] req@ffff895e4713ed80 x1855008961580160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079340.147543:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961580160/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079340.147554:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079340.147558:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079340.147562:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961580160:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079340.147569:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961580288:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079340.147589:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079340.147600:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079340/real 1769079340] req@ffff895e4713d880 x1855008961580288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079340.147631:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961580288/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079340.147640:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079340.147643:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079340.147646:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961580288:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079340.147652:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961580416:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079340.147671:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079340.147682:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079340/real 1769079340] req@ffff895e4713e300 x1855008961580416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079340.147693:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961580416/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079340.147701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079340.147705:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079340.147707:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961580416:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079340.339031:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079340.720737:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079341.360686:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079341.743869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079342.384215:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079342.767962:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079343.408279:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079343.791875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079344.431998:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079344.817049:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079345.264468:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079345.264478:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079345.264482:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079345.264492:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5027 00000100:00080000:3.0:1769079345.264501:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079345.264508:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079345.264545:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961580544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079345.264575:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079345.264578:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079345.264580:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079345.264584:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5027 00000100:00080000:3.0:1769079345.264588:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079345.264592:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079345.264601:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079345.264613:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961580672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079345.264622:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079345.264625:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079345.264627:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079345.264631:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5027 00000100:00080000:3.0:1769079345.264647:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079345.264651:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079345.264656:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079345.264677:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961580800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079345.264776:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961580544:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079345.264835:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079345.264871:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079345/real 1769079345] req@ffff895e453d5500 x1855008961580544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079345.264914:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961580544/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079345.264923:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079345.264927:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079345.264931:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961580544:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079345.264938:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961580672:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079345.264960:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079345.264972:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079345/real 1769079345] req@ffff895e453d4e00 x1855008961580672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079345.264982:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961580672/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079345.264990:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079345.264993:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079345.265009:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961580672:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079345.265016:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961580800:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079345.265034:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079345.265045:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079345/real 1769079345] req@ffff895e453d4a80 x1855008961580800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079345.265056:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961580800/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079345.265075:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079345.265078:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079345.265082:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961580800:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079345.456331:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079345.839962:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079346.480044:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079346.863937:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079347.504008:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079347.889484:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079348.528010:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079348.912002:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079349.553212:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079349.937015:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079350.383873:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079350.383885:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079350.383889:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079350.383899:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5032 00000100:00080000:3.0:1769079350.383921:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079350.383928:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079350.383968:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961580928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079350.384050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079350.384053:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079350.384055:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079350.384059:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5032 00000100:00080000:3.0:1769079350.384063:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079350.384067:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079350.384077:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079350.384090:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961581056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079350.384099:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961580928:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769079350.384102:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079350.384120:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079350.384122:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079350.384127:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5032 00000100:00080000:3.0:1769079350.384132:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079350.384136:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079350.384141:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079350.384153:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961581184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769079350.384162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079350.384202:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079350/real 1769079350] req@ffff895e453d7480 x1855008961580928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079350.384220:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961580928/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079350.384229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079350.384233:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079350.384236:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961580928:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079350.384259:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961581056:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079350.384279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079350.384291:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079350/real 1769079350] req@ffff895e453d5f80 x1855008961581056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079350.384302:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961581056/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079350.384322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079350.384325:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079350.384329:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961581056:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079350.384334:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961581184:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079350.384353:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079350.384362:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079350/real 1769079350] req@ffff895e453d5c00 x1855008961581184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079350.384373:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961581184/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079350.384381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079350.384384:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079350.384387:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961581184:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079350.576329:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079350.961677:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079351.602693:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079351.983895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079352.624604:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079353.008900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079353.651234:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079354.032097:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079354.671941:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079355.056327:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079355.503990:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079355.504001:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079355.504006:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079355.504016:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5038 00000100:00080000:1.0:1769079355.504026:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079355.504033:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079355.504070:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961581312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079355.504105:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079355.504109:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079355.504111:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079355.504115:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5038 00000100:00080000:1.0:1769079355.504119:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079355.504123:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079355.504134:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079355.504147:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961581440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079355.504153:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961581312:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769079355.504178:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079355.504182:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079355.504185:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079355.504190:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5038 00000100:00080000:1.0:1769079355.504194:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079355.504199:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079355.504204:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079355.504216:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961581568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769079355.504386:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079355.504427:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079355/real 1769079355] req@ffff895e4713f100 x1855008961581312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079355.504447:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961581312/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079355.504471:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079355.504475:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079355.504479:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961581312:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079355.504508:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961581440:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079355.504532:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079355.504544:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079355/real 1769079355] req@ffff895e4713ea00 x1855008961581440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079355.504563:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961581440/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079355.504571:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079355.504575:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079355.504578:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961581440:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079355.504584:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961581568:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079355.504603:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079355.504614:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079355/real 1769079355] req@ffff895e4713ce00 x1855008961581568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079355.504625:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961581568/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079355.504632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079355.504635:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079355.504638:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961581568:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079355.696057:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079356.079938:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079356.720012:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079357.103871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079357.744074:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079358.127926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079358.768003:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079359.151959:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079359.792901:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079360.176076:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079360.629068:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079360.629078:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079360.629083:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079360.629093:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5043 00000100:00080000:3.0:1769079360.629105:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079360.629112:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079360.629150:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961581696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079360.629184:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079360.629187:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079360.629189:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079360.629193:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5043 00000100:00080000:3.0:1769079360.629197:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079360.629201:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079360.629210:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079360.629223:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961581824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079360.629249:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079360.629252:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079360.629254:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079360.629257:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5043 00000100:00080000:3.0:1769079360.629260:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079360.629264:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079360.629268:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079360.629276:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961581952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079360.629396:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961581696:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079360.629461:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079360.629513:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079360/real 1769079360] req@ffff895e453d7800 x1855008961581696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079360.629534:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961581696/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079360.629544:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079360.629547:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079360.629551:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961581696:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079360.629575:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961581824:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079360.629600:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079360.629612:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079360/real 1769079360] req@ffff895e453d5c00 x1855008961581824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079360.629641:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961581824/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079360.629650:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079360.629653:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079360.629655:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961581824:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079360.629662:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961581952:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079360.629682:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079360.629692:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079360/real 1769079360] req@ffff895e453d5f80 x1855008961581952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079360.629703:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961581952/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079360.629710:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079360.629713:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079360.629716:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961581952:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079360.819269:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079361.200185:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079361.840324:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079362.223884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079362.864394:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079363.247939:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079363.888349:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079364.271866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079364.912203:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079365.295861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079365.743955:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079365.743967:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079365.743972:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079365.743981:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5048 00000100:00080000:3.0:1769079365.743991:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079365.743998:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079365.744037:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961582080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079365.744071:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079365.744074:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079365.744076:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079365.744081:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5048 00000100:00080000:3.0:1769079365.744085:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079365.744088:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079365.744110:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079365.744124:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961582208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079365.744134:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079365.744136:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079365.744138:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079365.744141:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5048 00000100:00080000:3.0:1769079365.744144:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079365.744148:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079365.744152:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079365.744160:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961582336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079365.744332:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961582080:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079365.744391:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079365.744427:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079365/real 1769079365] req@ffff895e453d7480 x1855008961582080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079365.744443:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961582080/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079365.744452:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079365.744456:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079365.744460:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961582080:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079365.744466:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961582208:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079365.744487:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079365.744498:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079365/real 1769079365] req@ffff895e453d4a80 x1855008961582208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079365.744509:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961582208/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079365.744517:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079365.744520:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079365.744523:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961582208:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079365.744528:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961582336:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079365.744545:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079365.744555:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079365/real 1769079365] req@ffff895e453d4e00 x1855008961582336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079365.744572:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961582336/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079365.744580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079365.744583:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079365.744586:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961582336:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079365.942062:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079366.321280:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079366.959989:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079367.344038:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079367.983957:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079368.383236:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079369.007969:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079369.391853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079370.032012:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079370.416620:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079370.864281:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079370.864291:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079370.864295:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079370.864305:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5053 00000100:00080000:3.0:1769079370.864315:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079370.864322:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079370.864356:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961582464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079370.864388:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079370.864391:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079370.864393:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079370.864396:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5053 00000100:00080000:3.0:1769079370.864414:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079370.864418:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079370.864426:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079370.864438:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961582592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079370.864448:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079370.864451:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079370.864453:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079370.864457:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5053 00000100:00080000:3.0:1769079370.864460:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079370.864464:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079370.864468:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079370.864477:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961582720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079370.864561:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961582464:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079370.864632:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079370.864669:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079370/real 1769079370] req@ffff895e453d5500 x1855008961582464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079370.864685:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961582464/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079370.864694:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079370.864698:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079370.864701:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961582464:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079370.864707:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961582592:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079370.864729:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079370.864741:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079370/real 1769079370] req@ffff895e453d7b80 x1855008961582592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079370.864787:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961582592/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079370.864795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079370.864798:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079370.864801:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961582592:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079370.864806:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961582720:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079370.864826:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079370.864836:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079370/real 1769079370] req@ffff895e453d7100 x1855008961582720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079370.864848:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961582720/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079370.864855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079370.864857:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079370.864860:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961582720:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079371.057243:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079371.443583:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079372.079962:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079372.464125:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079373.104543:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079373.487872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079374.128466:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079374.511876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079375.151918:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079375.535868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079375.984900:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079375.984916:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079375.984922:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079375.984937:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5058 00000100:00080000:3.0:1769079375.984955:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079375.984965:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079375.985015:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961582848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079375.985063:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079375.985090:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079375.985092:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079375.985097:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5058 00000100:00080000:3.0:1769079375.985102:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079375.985106:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079375.985118:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079375.985134:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961582976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079375.985144:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079375.985147:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079375.985149:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079375.985152:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5058 00000100:00080000:3.0:1769079375.985156:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079375.985160:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079375.985165:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079375.985176:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961583104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1769079375.988285:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961582848:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079375.988439:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079375.988515:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079375/real 1769079375] req@ffff895e453d4700 x1855008961582848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079375.988540:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961582848/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079375.988553:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079375.988559:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079375.988564:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961582848:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079375.988575:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961582976:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079375.988600:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079375.988613:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079375/real 1769079375] req@ffff895e453d5180 x1855008961582976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079375.988625:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961582976/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079375.988634:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079375.988638:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079375.988641:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961582976:192.168.204.117@tcp:38:kworker.0 00000100:00100000:1.0:1769079375.988649:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961583104:192.168.204.117@tcp:38:kworker.0 00000800:00000100:1.0:1769079375.988681:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769079375.988692:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079375/real 1769079375] req@ffff895e453d6300 x1855008961583104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1769079375.988726:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961583104/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1769079375.988734:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1769079375.988738:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:1.0:1769079375.988741:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961583104:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079376.177009:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079376.564659:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079377.201572:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079377.590861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079378.224560:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079378.608245:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079379.248271:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079379.631976:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079380.272201:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079380.658079:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079381.104316:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079381.104328:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079381.104334:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079381.104344:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5063 00000100:00080000:1.0:1769079381.104357:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079381.104366:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079381.104416:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961583232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079381.104451:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079381.104455:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079381.104457:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079381.104461:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5063 00000100:00080000:1.0:1769079381.104465:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079381.104487:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079381.104496:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079381.104508:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961583360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079381.104516:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079381.104519:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079381.104521:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079381.104524:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5063 00000100:00080000:1.0:1769079381.104527:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079381.104531:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079381.104535:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079381.104543:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961583488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079381.104634:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961583232:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079381.104704:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079381.104780:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079381/real 1769079381] req@ffff895e4713d500 x1855008961583232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079381.104797:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961583232/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079381.104806:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079381.104822:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079381.104825:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961583232:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079381.104832:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961583360:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079381.104864:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079381.104879:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079381/real 1769079381] req@ffff895e4713f480 x1855008961583360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079381.104909:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961583360/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079381.104917:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079381.104920:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079381.104923:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961583360:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079381.104929:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961583488:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079381.104947:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079381.104959:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079381/real 1769079381] req@ffff895e4713ca80 x1855008961583488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079381.104969:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961583488/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079381.104976:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079381.104979:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079381.104981:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961583488:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079381.296405:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079381.679849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079382.320676:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079382.704040:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079383.345296:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079383.729922:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079384.368868:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079384.753069:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079385.391933:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079385.777017:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079386.223913:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079386.223923:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079386.223927:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079386.223937:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5068 00000100:00080000:3.0:1769079386.223947:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079386.223954:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079386.223986:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961583616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079386.224031:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079386.224035:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079386.224037:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079386.224042:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5068 00000100:00080000:3.0:1769079386.224046:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079386.224049:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079386.224059:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079386.224071:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961583744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079386.224079:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079386.224082:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079386.224084:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079386.224087:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5068 00000100:00100000:0.0:1769079386.224090:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961583616:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769079386.224090:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079386.224093:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079386.224097:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079386.224115:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961583872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769079386.224153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079386.224196:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079386/real 1769079386] req@ffff895e453d5880 x1855008961583616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079386.224212:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961583616/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079386.224221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079386.224225:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079386.224229:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961583616:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079386.224236:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961583744:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079386.224270:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079386.224283:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079386/real 1769079386] req@ffff895e453d4380 x1855008961583744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079386.224294:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961583744/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079386.224302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079386.224313:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079386.224317:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961583744:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079386.224348:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961583872:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079386.224368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079386.224389:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079386/real 1769079386] req@ffff895e453d6680 x1855008961583872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079386.224399:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961583872/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079386.224418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079386.224421:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079386.224424:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961583872:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079386.416022:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079386.800705:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079387.440029:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079387.823887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079388.463983:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079388.847883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079389.490772:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079389.871926:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079390.512090:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079390.898192:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079391.344098:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079391.344109:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079391.344113:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079391.344123:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5073 00000100:00080000:1.0:1769079391.344135:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079391.344143:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079391.344181:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961584000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079391.344213:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079391.344216:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079391.344218:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079391.344230:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5073 00000100:00080000:1.0:1769079391.344234:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079391.344238:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079391.344247:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079391.344259:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961584128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079391.344268:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079391.344271:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079391.344273:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079391.344276:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5073 00000100:00080000:1.0:1769079391.344280:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079391.344284:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079391.344288:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079391.344317:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961584256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079391.344544:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961584000:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079391.344623:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079391.344673:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079391/real 1769079391] req@ffff895e4713f800 x1855008961584000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079391.344692:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961584000/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079391.344701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079391.344705:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079391.344709:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961584000:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079391.344716:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961584128:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079391.344736:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079391.345100:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079391/real 1769079391] req@ffff895e4713dc00 x1855008961584128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079391.345115:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961584128/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079391.345138:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079391.345142:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079391.345146:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961584128:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079391.345152:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961584256:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079391.345176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079391.345189:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079391/real 1769079391] req@ffff895e4713df80 x1855008961584256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079391.345199:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961584256/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079391.345206:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079391.345209:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079391.345212:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961584256:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079391.536539:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079391.919911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079392.560022:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079392.943883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079393.583970:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079393.967855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079394.610125:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079394.991901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079395.635379:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079396.015878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079396.463876:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079396.463887:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079396.463892:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079396.463901:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5079 00000100:00080000:3.0:1769079396.463936:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079396.463944:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079396.463987:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961584384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079396.464016:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079396.464019:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079396.464021:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079396.464024:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5079 00000100:00080000:3.0:1769079396.464028:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079396.464032:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079396.464040:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079396.464052:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961584512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079396.464062:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079396.464065:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079396.464066:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079396.464070:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5079 00000100:00080000:3.0:1769079396.464073:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079396.464077:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079396.464082:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079396.464091:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961584640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079396.464278:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961584384:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079396.464388:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961584512:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079396.464412:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961584640:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079396.656181:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079397.039890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079397.681629:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079398.063874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079398.704510:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079399.087905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769079399.546370:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769079399.546431:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769079399.546449:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079399.548476:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079399.548495:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:2.0:1769079399.551268:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079396/real 1769079399] req@ffff895e453d6d80 x1855008961584384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079399.551306:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961584384/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079399.551317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079399.551321:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079399.551325:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961584384:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079399.551333:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079396/real 1769079399] req@ffff895e453d6680 x1855008961584512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079399.551345:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961584512/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079399.551353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079399.551356:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079399.551359:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961584512:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079399.551367:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079396/real 1769079399] req@ffff895e453d4380 x1855008961584640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079399.551379:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961584640/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079399.551386:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079399.551407:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079399.551411:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961584640:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079399.728009:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079400.111847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079400.751985:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079401.139802:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079401.584278:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079401.584312:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079401.584318:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079401.584333:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5084 00000100:00080000:3.0:1769079401.584351:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079401.584363:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079401.584410:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961584768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079401.584448:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079401.584452:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079401.584454:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079401.584458:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5084 00000100:00080000:3.0:1769079401.584463:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079401.584467:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079401.584479:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079401.584492:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961584896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079401.584502:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079401.584815:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079401.584818:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079401.584823:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5084 00000100:00080000:3.0:1769079401.584836:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079401.584841:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079401.584847:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079401.584860:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961585024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079401.587385:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961584768:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079401.587477:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079401.587528:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079401/real 1769079401] req@ffff895e453d5880 x1855008961584768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079401.587549:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961584768/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079401.587560:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079401.587565:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079401.587569:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961584768:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079401.587577:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961584896:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079401.587599:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079401.587612:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079401/real 1769079401] req@ffff895e453d6300 x1855008961584896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079401.587621:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961584896/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079401.587629:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079401.587632:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079401.587635:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961584896:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079401.587856:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961585024:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079401.587896:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079401.587908:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079401/real 1769079401] req@ffff895e453d5180 x1855008961585024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079401.587921:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961585024/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079401.587929:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079401.587932:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079401.587935:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961585024:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079401.775981:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079402.160267:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079402.801023:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079403.183900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079403.824774:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079404.207880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079404.848123:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079405.233405:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079405.872029:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079406.256006:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079406.705214:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079406.705225:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079406.705229:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079406.705249:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5089 00000100:00080000:0.0:1769079406.705261:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079406.705268:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079406.705310:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961585152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079406.705341:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079406.705345:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079406.705347:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079406.705351:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5089 00000100:00080000:0.0:1769079406.705354:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079406.705358:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079406.705367:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079406.705388:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961585280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079406.705397:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079406.705401:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079406.705402:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079406.705405:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5089 00000100:00080000:0.0:1769079406.705421:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079406.705425:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079406.705430:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079406.705441:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961585408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079406.705617:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961585152:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079406.705679:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079406.705718:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079406/real 1769079406] req@ffff895e4383d880 x1855008961585152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079406.705734:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961585152/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079406.705794:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079406.705800:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079406.705804:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961585152:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079406.705811:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961585280:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079406.705834:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079406.705845:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079406/real 1769079406] req@ffff895e4383e680 x1855008961585280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079406.705868:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961585280/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079406.705876:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079406.705879:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079406.705881:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961585280:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079406.706644:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961585408:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079406.706673:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079406.706687:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079406/real 1769079406] req@ffff895e4383c700 x1855008961585408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079406.706699:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961585408/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079406.706708:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079406.706711:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079406.706714:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961585408:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079406.896306:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079407.279878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079407.930559:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079408.303877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079408.950256:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079409.327863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079409.968114:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079410.351913:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079410.993405:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079411.376415:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079411.823953:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079411.823966:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079411.823970:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079411.823986:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5094 00000100:00080000:2.0:1769079411.823997:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079411.824004:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079411.824043:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961585536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079411.824089:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079411.824092:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079411.824094:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079411.824098:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5094 00000100:00080000:2.0:1769079411.824102:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079411.824106:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079411.824115:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079411.824128:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961585664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079411.824136:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079411.824138:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079411.824140:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079411.824143:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5094 00000100:00080000:2.0:1769079411.824146:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079411.824149:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079411.824154:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079411.824162:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961585792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079411.824320:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961585536:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079411.824388:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079411.824425:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079411/real 1769079411] req@ffff895e44e2aa00 x1855008961585536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079411.824442:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961585536/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079411.824451:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079411.824454:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079411.824457:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961585536:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079411.824464:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961585664:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079411.824483:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079411.824494:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079411/real 1769079411] req@ffff895e44e2a300 x1855008961585664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079411.824504:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961585664/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079411.824517:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079411.824520:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079411.824523:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961585664:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079411.824528:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961585792:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079411.824561:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079411.824581:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079411/real 1769079411] req@ffff895e44e2b100 x1855008961585792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079411.824592:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961585792/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079411.824599:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079411.824602:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079411.824605:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961585792:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079412.016032:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079412.399982:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079413.040197:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079413.423987:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079414.066099:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079414.448433:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079415.088420:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079415.471880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079416.112727:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079416.496219:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079416.944099:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079416.944112:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079416.944117:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079416.944127:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5099 00000100:00080000:0.0:1769079416.944138:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079416.944145:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079416.944182:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961585920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079416.944226:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079416.944230:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079416.944232:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079416.944236:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5099 00000100:00080000:0.0:1769079416.944239:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079416.944243:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079416.944251:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079416.944262:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961586048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079416.944271:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079416.944274:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079416.944276:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079416.944279:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5099 00000100:00080000:0.0:1769079416.944281:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079416.944284:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079416.944287:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079416.944295:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961586176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079416.944368:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961585920:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079416.944424:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079416.944457:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079416/real 1769079416] req@ffff895e4383f100 x1855008961585920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079416.944484:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961585920/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079416.944493:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079416.944497:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079416.944500:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961585920:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079416.944507:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961586048:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079416.944524:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079416.944534:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079416/real 1769079416] req@ffff895e4383c380 x1855008961586048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079416.944545:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961586048/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079416.944552:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079416.944555:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079416.944558:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961586048:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079416.944562:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961586176:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079416.944574:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079416.944582:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079416/real 1769079416] req@ffff895e4383fb80 x1855008961586176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079416.944592:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961586176/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079416.944612:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079416.944616:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079416.944619:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961586176:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079417.136019:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079417.519895:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079418.160085:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079418.543888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079419.186152:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079419.567879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079420.211687:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079420.591892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079421.232068:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079421.615896:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079422.063993:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079422.064002:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079422.064005:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079422.064014:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5104 00000100:00080000:1.0:1769079422.064024:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079422.064029:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079422.064059:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961586304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079422.064077:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079422.064080:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079422.064083:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079422.064086:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5104 00000100:00080000:1.0:1769079422.064105:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079422.064109:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079422.064118:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079422.064128:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961586304:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769079422.064129:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961586432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079422.064138:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079422.064141:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079422.064143:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079422.064146:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5104 00000100:00080000:1.0:1769079422.064148:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079422.064151:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079422.064155:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079422.064164:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961586560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769079422.064183:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079422.064520:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079422/real 1769079422] req@ffff895e4713c380 x1855008961586304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079422.064537:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961586304/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079422.064546:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079422.064550:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079422.064554:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961586304:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079422.064577:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961586432:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079422.065811:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079422.065832:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079422/real 1769079422] req@ffff895e4713d180 x1855008961586432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079422.065846:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961586432/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079422.065855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079422.065858:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079422.065862:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961586432:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079422.065882:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961586560:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079422.065903:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079422.065914:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079422/real 1769079422] req@ffff895e4713df80 x1855008961586560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079422.065925:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961586560/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079422.065932:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079422.065935:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079422.065938:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961586560:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079422.256049:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079422.639955:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079423.283152:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079423.663860:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079424.304007:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079424.689133:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079425.328263:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079425.711927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079426.351957:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079426.735946:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079427.184144:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079427.184155:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079427.184160:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079427.184169:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5109 00000100:00080000:0.0:1769079427.184181:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079427.184187:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079427.184225:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961586688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079427.184256:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079427.184260:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079427.184262:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079427.184266:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5109 00000100:00080000:0.0:1769079427.184269:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079427.184273:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079427.184282:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079427.184294:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961586816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079427.184304:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079427.184307:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079427.184308:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079427.184312:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5109 00000100:00080000:0.0:1769079427.184316:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079427.184319:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079427.184338:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079427.184349:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961586944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079427.184433:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961586688:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079427.184515:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079427.184563:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079427/real 1769079427] req@ffff895e4383ce00 x1855008961586688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079427.184580:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961586688/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079427.184589:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079427.184593:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079427.184596:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961586688:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079427.184604:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961586816:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079427.184658:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079427.184670:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079427/real 1769079427] req@ffff895e4383f480 x1855008961586816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079427.184688:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961586816/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079427.184696:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079427.184699:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079427.184702:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961586816:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079427.184707:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961586944:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079427.184728:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079427.184738:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079427/real 1769079427] req@ffff895e4383c000 x1855008961586944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079427.184774:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961586944/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079427.184781:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079427.184783:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079427.184786:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961586944:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079427.377551:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079427.759905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079428.403693:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079428.783916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079429.423955:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079429.807879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079430.448006:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079430.832942:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079431.472003:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079431.855892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079432.304006:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079432.304018:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079432.304023:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079432.304033:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5114 00000100:00080000:0.0:1769079432.304053:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079432.304060:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079432.304100:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961587072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079432.304124:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079432.304127:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079432.304129:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079432.304132:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5114 00000100:00080000:0.0:1769079432.304136:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079432.304139:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079432.304148:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079432.304159:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961587200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079432.304167:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079432.304186:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079432.304188:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079432.304191:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5114 00000100:00080000:0.0:1769079432.304194:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079432.304198:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079432.304202:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079432.304211:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961587328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079432.304827:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961587072:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079432.304883:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079432.304916:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079432/real 1769079432] req@ffff895e4383ea00 x1855008961587072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079432.304931:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961587072/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079432.304952:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079432.304956:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079432.304959:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961587072:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079432.304966:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961587200:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079432.304984:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079432.305004:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079432/real 1769079432] req@ffff895e4383dc00 x1855008961587200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079432.305015:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961587200/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079432.305022:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079432.305025:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079432.305028:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961587200:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079432.305033:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961587328:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079432.305161:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079432.305172:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079432/real 1769079432] req@ffff895e4383ca80 x1855008961587328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079432.305183:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961587328/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079432.305191:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079432.305193:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079432.305196:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961587328:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079432.505266:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079432.879865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079433.520028:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079433.904282:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079434.549232:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079434.928351:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079435.568065:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079435.951916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079436.592165:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079436.975873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079437.431069:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079437.431081:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079437.431085:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079437.431095:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5119 00000100:00080000:1.0:1769079437.431107:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079437.431114:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079437.431153:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961587456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079437.431231:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961587456:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079437.431292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:1.0:1769079437.431323:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079437.431327:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079437.431328:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079437.431332:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5119 00000100:00080000:1.0:1769079437.431336:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079437.431340:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079437.431348:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079437.431359:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961587584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079437.431421:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079437/real 1769079437] req@ffff895e4713dc00 x1855008961587456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079437.431437:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961587456/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079437.431457:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079437.431461:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079437.431465:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961587456:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079437.431489:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961587584:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079437.431511:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079437.431523:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079437/real 1769079437] req@ffff895e4713f800 x1855008961587584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079437.431534:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961587584/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079437.431542:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079437.431545:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079437.431547:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961587584:192.168.204.117@tcp:38:kworker.0 00000100:00080000:1.0:1769079437.431872:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079437.431967:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079437.431982:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079437.431987:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5119 00000100:00080000:1.0:1769079437.431991:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079437.431995:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079437.432001:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079437.432014:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961587712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079437.432066:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961587712:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079437.432092:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079437.432164:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079437/real 1769079437] req@ffff895e4713ca80 x1855008961587712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079437.432177:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961587712/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079437.432185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079437.432189:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079437.432192:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961587712:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079437.618460:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079437.999879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079438.640505:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079439.024120:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079439.664123:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079440.047878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079440.688517:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079441.071882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079441.713895:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079442.095827:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079442.544342:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079442.544353:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079442.544370:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079442.544381:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5125 00000100:00080000:0.0:1769079442.544392:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079442.544399:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079442.544440:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961587840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079442.544523:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961587840:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079442.544585:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769079442.544617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079442.544621:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079442.544634:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079442.544638:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5125 00000100:00080000:0.0:1769079442.544642:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079442.544647:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079442.544655:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079442.544668:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961587968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079442.545637:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079442/real 1769079442] req@ffff895e4383d500 x1855008961587840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079442.545658:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961587840/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079442.545668:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079442.545673:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079442.545676:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961587840:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079442.545839:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961587968:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079442.545879:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079442.545906:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079442/real 1769079442] req@ffff895e4383f800 x1855008961587968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079442.545918:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961587968/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079442.545935:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079442.545939:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079442.545942:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961587968:192.168.204.117@tcp:38:kworker.0 00000100:00080000:0.0:1769079442.546290:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079442.546296:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079442.546299:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079442.546415:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5125 00000100:00080000:0.0:1769079442.546422:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079442.546428:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079442.546437:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079442.546457:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961588096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079442.546506:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961588096:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079442.546619:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079442.546686:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079442/real 1769079442] req@ffff895e4383e300 x1855008961588096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079442.546817:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961588096/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079442.546826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079442.546830:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079442.546833:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961588096:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079442.737006:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079443.119887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079443.760607:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079444.143850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079444.788062:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079445.167869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079445.807951:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079446.192053:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079446.835863:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079447.215901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079447.663872:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079447.663883:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079447.663887:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079447.663896:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5130 00000100:00080000:0.0:1769079447.663907:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079447.663915:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079447.663973:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961588224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079447.664060:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961588224:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079447.664143:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769079447.664182:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079447.664185:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079447.664187:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079447.664192:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5130 00000100:00080000:0.0:1769079447.664196:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079447.664200:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079447.664209:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079447.664222:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961588352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079447.664496:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079447/real 1769079447] req@ffff895e4383e300 x1855008961588224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079447.664514:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961588224/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079447.664524:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079447.664536:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079447.664540:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961588224:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079447.664567:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961588352:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079447.664606:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079447.664619:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079447/real 1769079447] req@ffff895e4383f800 x1855008961588352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079447.664630:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961588352/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079447.664638:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079447.664641:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079447.664644:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961588352:192.168.204.117@tcp:38:kworker.0 00000100:00080000:0.0:1769079447.664894:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079447.664899:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079447.664901:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079447.664906:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5130 00000100:00080000:0.0:1769079447.664910:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079447.664914:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079447.664921:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079447.664936:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961588480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079447.665036:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961588480:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079447.665060:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079447.665071:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079447/real 1769079447] req@ffff895e4383d500 x1855008961588480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079447.665082:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961588480/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079447.665090:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079447.665093:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079447.665096:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961588480:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079447.856044:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079448.239941:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079448.879984:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079449.263986:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079449.904342:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079450.289523:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079450.928182:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079451.311880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079451.951952:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079452.335880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079452.785420:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079452.785431:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079452.785436:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079452.785447:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5135 00000100:00080000:3.0:1769079452.785457:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079452.785464:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079452.785500:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961588608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079452.785534:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079452.785538:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079452.785540:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079452.785544:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5135 00000100:00080000:3.0:1769079452.785547:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079452.785551:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079452.785559:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079452.785572:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961588736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079452.785580:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079452.785582:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079452.785584:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079452.785587:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5135 00000100:00080000:3.0:1769079452.785590:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079452.785594:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079452.785613:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079452.785623:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961588864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079452.786167:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961588608:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079452.786242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079452.786278:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079452/real 1769079452] req@ffff895e453d4700 x1855008961588608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079452.786294:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961588608/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079452.786304:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079452.786307:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079452.786311:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961588608:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079452.786318:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961588736:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079452.786337:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079452.786348:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079452/real 1769079452] req@ffff895e453d7100 x1855008961588736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079452.786359:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961588736/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079452.786367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079452.786370:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079452.786372:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961588736:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079452.786378:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961588864:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079452.786393:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079452.786403:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079452/real 1769079452] req@ffff895e453d7b80 x1855008961588864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079452.786422:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961588864/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079452.786429:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079452.786431:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079452.786434:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961588864:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079452.976000:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079453.359869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079453.999993:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079454.383850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079455.024901:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079455.409772:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079456.059611:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079456.431880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079457.073016:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079457.456878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079457.904678:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079457.904694:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079457.904708:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079457.904723:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5140 00000100:00080000:3.0:1769079457.904737:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079457.904765:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079457.904816:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961588992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079457.904892:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961588992:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079457.904973:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769079457.906386:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079457.906393:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079457.906397:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079457.906406:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5140 00000100:00080000:3.0:1769079457.906422:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079457.906429:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079457.906443:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079457.906472:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961589120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079457.906487:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079457.906490:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079457.906492:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079457.906496:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5140 00000100:00080000:3.0:1769079457.906518:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079457.906522:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079457.906527:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079457.906538:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961589248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079457.906819:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079457/real 1769079457] req@ffff895e453d5500 x1855008961588992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079457.906842:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961588992/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079457.906853:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079457.906857:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079457.906861:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961588992:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079457.906898:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961589120:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079457.906947:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079457.906961:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079457/real 1769079457] req@ffff895e453d4e00 x1855008961589120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079457.906972:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961589120/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079457.908242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079457.908250:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079457.908256:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961589120:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079457.908269:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961589248:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079457.908317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079457.908335:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079457/real 1769079457] req@ffff895e453d4a80 x1855008961589248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079457.908347:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961589248/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079457.908354:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079457.908358:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079457.908361:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961589248:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079458.105140:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079458.482540:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079459.121788:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079459.504131:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079460.144557:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079460.527949:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079461.167920:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079461.551872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079462.192633:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079462.577311:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079463.024448:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079463.024459:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079463.024463:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079463.024473:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5145 00000100:00080000:0.0:1769079463.024483:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079463.024491:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079463.024530:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961589376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079463.024559:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079463.024562:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079463.024564:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079463.024568:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5145 00000100:00080000:0.0:1769079463.024572:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079463.024576:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079463.024598:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079463.024613:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961589504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079463.024623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079463.024626:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079463.024646:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1769079463.024653:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961589376:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769079463.024654:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5145 00000100:00080000:0.0:1769079463.024661:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079463.024668:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079463.024678:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079463.024694:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961589632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079463.024862:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961589504:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079463.024905:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961589632:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079463.218652:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079463.599892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079464.242656:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079464.624548:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079465.264117:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079465.647898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769079466.096064:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769079466.096111:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769079466.096131:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079466.096173:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079466.096181:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:2.0:1769079466.096336:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079463/real 1769079466] req@ffff895e4383ea00 x1855008961589632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079466.096382:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961589632/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079466.096394:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079466.096399:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079466.096403:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961589632:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079466.096411:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079463/real 1769079466] req@ffff895e4383ca80 x1855008961589376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079466.096422:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961589376/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079466.096429:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079466.096433:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079466.096436:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961589376:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079466.096442:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079463/real 1769079466] req@ffff895e4383dc00 x1855008961589504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079466.096453:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961589504/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079466.096460:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079466.096463:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079466.096466:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961589504:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079466.288952:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079466.671989:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079467.312814:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079467.696504:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079468.153175:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079468.153188:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079468.153193:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079468.153203:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5150 00000100:00080000:3.0:1769079468.153214:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079468.153221:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079468.153275:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961589760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079468.153308:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079468.153312:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079468.153314:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079468.153318:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5150 00000100:00080000:3.0:1769079468.153322:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079468.153326:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079468.153335:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079468.153348:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961589888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079468.153411:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961589760:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079468.153478:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079468.153521:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079468/real 1769079468] req@ffff895e453d7480 x1855008961589760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079468.153539:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961589760/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079468.153549:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079468.153559:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079468.153562:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961589760:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769079468.153835:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079468.153853:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079468.153855:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079468.153860:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5150 00000100:00080000:3.0:1769079468.153864:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079468.153868:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079468.153874:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079468.153887:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961590016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079468.154238:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961589888:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079468.154284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079468.154300:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079468/real 1769079468] req@ffff895e453d5f80 x1855008961589888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079468.154314:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961589888/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079468.154324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079468.154328:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079468.154330:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961589888:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079468.154336:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961590016:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079468.154355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079468.154377:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079468/real 1769079468] req@ffff895e453d5c00 x1855008961590016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079468.154415:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961590016/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079468.154422:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079468.154425:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079468.154427:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961590016:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079468.336207:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079468.719866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079469.359956:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079469.746934:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079470.384015:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079470.767866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079471.408190:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079471.793042:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079472.431966:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079472.815886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079473.263906:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079473.263918:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079473.263922:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079473.264923:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5155 00000100:00080000:3.0:1769079473.264935:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079473.265005:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079473.265044:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961590144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079473.265131:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961590144:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079473.265196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769079473.265238:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079473.265242:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079473.265244:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079473.265248:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5155 00000100:00080000:3.0:1769079473.265253:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079473.265257:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079473.265266:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079473.265280:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961590272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079473.269128:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079473/real 1769079473] req@ffff895e453d7800 x1855008961590144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079473.269155:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961590144/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079473.269165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079473.269170:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079473.269173:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961590144:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079473.270180:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961590272:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079473.270346:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079473.270455:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079473/real 1769079473] req@ffff895e453d5c00 x1855008961590272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079473.270472:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961590272/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079473.270501:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079473.270506:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079473.270510:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961590272:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769079473.273128:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079473.273137:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079473.273142:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079473.273150:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5155 00000100:00080000:3.0:1769079473.273160:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079473.273167:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079473.273179:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079473.273442:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961590400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079473.273495:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961590400:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079473.273864:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079473.273892:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079473/real 1769079473] req@ffff895e453d5f80 x1855008961590400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079473.273908:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961590400/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079473.273916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079473.273921:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079473.273924:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961590400:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079473.457623:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079473.839953:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079474.480542:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079474.864375:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079475.503984:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079475.889820:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079476.528887:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079476.912471:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079477.552710:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079477.935882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079478.390952:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079478.390964:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079478.390969:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079478.390978:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5160 00000100:00080000:0.0:1769079478.390989:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079478.390997:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079478.391036:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961590528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079478.391072:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079478.391076:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079478.391078:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079478.391082:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5160 00000100:00080000:0.0:1769079478.391102:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079478.391108:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079478.391117:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079478.391130:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961590656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079478.391140:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079478.391142:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079478.391144:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079478.391147:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5160 00000100:00080000:0.0:1769079478.391151:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079478.391155:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079478.391160:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079478.391170:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961590784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079478.391389:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961590528:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079478.391451:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079478.391486:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079478/real 1769079478] req@ffff895e4383c000 x1855008961590528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079478.391502:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961590528/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079478.391511:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079478.391521:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079478.391525:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961590528:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079478.391532:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961590656:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079478.391556:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079478.391568:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079478/real 1769079478] req@ffff895e4383f480 x1855008961590656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079478.391580:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961590656/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079478.391588:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079478.391591:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079478.391594:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961590656:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079478.391599:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961590784:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079478.391634:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079478.391644:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079478/real 1769079478] req@ffff895e4383ce00 x1855008961590784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079478.391656:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961590784/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079478.391674:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079478.391677:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079478.391680:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961590784:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079478.577855:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079478.960200:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079479.600024:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079479.983944:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079480.624328:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079481.007947:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079481.650349:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079482.031876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079482.673101:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079483.056326:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079483.508121:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079483.508133:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079483.508138:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079483.508147:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5166 00000100:00080000:0.0:1769079483.508157:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079483.508163:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079483.508201:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961590912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079483.508244:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079483.508246:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079483.508248:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079483.508251:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5166 00000100:00080000:0.0:1769079483.508254:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079483.508257:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079483.508264:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079483.508273:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961591040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079483.508279:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079483.508281:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079483.508282:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079483.508284:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5166 00000100:00080000:0.0:1769079483.508285:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079483.508287:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079483.508290:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079483.508295:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961591168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079483.508478:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961590912:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079483.508527:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079483.508561:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079483/real 1769079483] req@ffff895e4383fb80 x1855008961590912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079483.508578:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961590912/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079483.508588:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079483.508590:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079483.508593:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961590912:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079483.508597:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961591040:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079483.508613:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079483.508620:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079483/real 1769079483] req@ffff895e4383c380 x1855008961591040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079483.508627:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961591040/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079483.508632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079483.508633:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079483.508635:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961591040:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079483.508638:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961591168:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079483.508647:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079483.508651:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079483/real 1769079483] req@ffff895e4383f100 x1855008961591168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079483.508664:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961591168/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079483.508681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079483.508684:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079483.508688:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961591168:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079483.697542:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079484.079882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079484.720467:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079485.104026:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079485.744380:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079486.127893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079486.773106:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079487.152502:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079487.792119:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079488.176911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079488.623871:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079488.623898:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079488.623903:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079488.623912:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5171 00000100:00080000:0.0:1769079488.623923:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079488.623930:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079488.623965:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961591296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079488.623996:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079488.623999:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079488.624001:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079488.624005:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5171 00000100:00080000:0.0:1769079488.624008:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079488.624012:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079488.624020:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079488.624031:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961591424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079488.624039:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079488.624042:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079488.624044:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079488.624047:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5171 00000100:00080000:0.0:1769079488.624051:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079488.624054:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079488.624058:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079488.624074:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961591552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079488.624181:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961591296:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079488.624239:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079488.624276:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079488/real 1769079488] req@ffff895e4383c700 x1855008961591296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079488.624294:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961591296/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079488.624303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079488.624307:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079488.624311:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961591296:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079488.624319:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961591424:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079488.624339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079488.624351:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079488/real 1769079488] req@ffff895e4383e680 x1855008961591424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079488.624362:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961591424/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079488.624381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079488.624386:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079488.624389:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961591424:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079488.624395:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961591552:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079488.624414:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079488.624424:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079488/real 1769079488] req@ffff895e4383d880 x1855008961591552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079488.624433:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961591552/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079488.624440:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079488.624442:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079488.624445:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961591552:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079488.816059:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079489.202677:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079489.840115:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079490.227110:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079490.863967:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079491.247888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079491.888300:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079492.272562:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079492.920799:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079493.295875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079493.746068:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079493.746078:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079493.746083:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079493.746093:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5176 00000100:00080000:3.0:1769079493.746105:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079493.746113:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079493.746153:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961591680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079493.746189:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079493.746192:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079493.746194:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079493.746198:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5176 00000100:00080000:3.0:1769079493.746201:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079493.746204:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079493.746229:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079493.746254:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961591808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079493.746271:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079493.746275:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079493.746277:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079493.746281:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5176 00000100:00080000:3.0:1769079493.746285:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079493.746289:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079493.746294:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079493.746303:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961591936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079493.746576:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961591680:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079493.746637:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079493.746672:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079493/real 1769079493] req@ffff895e453d7480 x1855008961591680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079493.746688:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961591680/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079493.746697:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079493.746708:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079493.746711:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961591680:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079493.746718:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961591808:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079493.746738:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079493.746774:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079493/real 1769079493] req@ffff895e453d4a80 x1855008961591808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079493.746785:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961591808/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079493.746792:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079493.746795:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079493.746798:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961591808:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079493.746803:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961591936:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079493.746834:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079493.746844:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079493/real 1769079493] req@ffff895e453d4e00 x1855008961591936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079493.746854:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961591936/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079493.746861:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079493.746863:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079493.746866:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961591936:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079493.936047:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079494.319877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079494.959965:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079495.344150:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079495.984359:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079496.368328:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079497.008015:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079497.392146:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079498.031960:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079498.416883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079498.863910:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079498.863920:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079498.863924:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079498.863934:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5181 00000100:00080000:0.0:1769079498.863944:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079498.863951:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079498.863989:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961592064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079498.864015:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079498.864018:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079498.864020:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079498.864025:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5181 00000100:00080000:0.0:1769079498.864028:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079498.864031:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079498.864040:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079498.864052:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961592192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079498.864061:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079498.864088:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079498.864089:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079498.864093:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5181 00000100:00080000:0.0:1769079498.864098:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079498.864102:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079498.864106:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079498.864118:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961592320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079498.864621:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961592064:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079498.864691:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079498.864726:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079498/real 1769079498] req@ffff895e4383ed80 x1855008961592064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079498.864765:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961592064/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079498.864775:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079498.864779:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079498.864783:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961592064:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079498.864790:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961592192:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079498.864811:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079498.864824:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079498/real 1769079498] req@ffff895e4383df80 x1855008961592192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079498.864835:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961592192/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079498.864843:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079498.864846:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079498.864994:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961592192:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079498.865003:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961592320:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079498.865026:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079498.865037:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079498/real 1769079498] req@ffff895e4383d180 x1855008961592320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079498.865056:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961592320/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079498.865061:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079498.865063:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079498.865065:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961592320:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079499.059563:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079499.439880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079500.079989:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079500.467679:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079501.104047:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079501.487863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079502.128675:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079502.511927:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079503.156933:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079503.535877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079503.983913:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079503.983924:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079503.983939:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079503.983948:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5186 00000100:00080000:3.0:1769079503.983960:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079503.983966:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079503.984004:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961592448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079503.984037:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079503.984040:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079503.984042:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079503.984045:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5186 00000100:00080000:3.0:1769079503.984066:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079503.984070:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079503.984079:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079503.984093:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961592576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079503.984101:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079503.984118:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079503.984121:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079503.984132:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5186 00000100:00080000:3.0:1769079503.984136:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079503.984140:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079503.984145:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079503.984155:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961592704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079503.985142:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961592448:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079503.985227:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079503.985276:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079503/real 1769079503] req@ffff895e453d5500 x1855008961592448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079503.985294:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961592448/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079503.985303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079503.985307:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079503.985310:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961592448:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079503.985317:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961592576:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079503.985360:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079503.985371:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079503/real 1769079503] req@ffff895e453d7b80 x1855008961592576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079503.985398:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961592576/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079503.985407:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079503.985410:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079503.985414:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961592576:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079503.985419:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961592704:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079503.985439:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079503.985450:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079503/real 1769079503] req@ffff895e453d7100 x1855008961592704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079503.985460:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961592704/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079503.985467:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079503.985469:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079503.985472:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961592704:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079504.176468:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079504.560589:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079505.199984:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079505.583936:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079506.225508:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079506.607850:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079507.248365:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079507.633619:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079508.274083:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079508.655955:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079509.103912:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079509.103924:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079509.103928:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079509.103938:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5191 00000100:00080000:1.0:1769079509.103950:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079509.103957:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079509.104000:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961592832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079509.104048:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079509.104051:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079509.104053:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079509.104057:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5191 00000100:00080000:1.0:1769079509.104061:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079509.104065:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079509.104073:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079509.104086:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961592960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079509.104094:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079509.104097:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079509.104099:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079509.104102:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5191 00000100:00080000:1.0:1769079509.104105:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079509.104108:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079509.104112:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079509.104120:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961593088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079509.104154:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961592832:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079509.104215:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079509.104262:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079509/real 1769079509] req@ffff895e4713f480 x1855008961592832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079509.104291:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961592832/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079509.104301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079509.104305:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079509.104310:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961592832:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079509.104317:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961592960:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079509.104339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079509.104352:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079509/real 1769079509] req@ffff895e4713d500 x1855008961592960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079509.104364:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961592960/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079509.104372:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079509.104375:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079509.104378:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961592960:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079509.104383:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961593088:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079509.104401:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079509.104411:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079509/real 1769079509] req@ffff895e4713ce00 x1855008961593088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079509.104422:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961593088/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079509.104441:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079509.104445:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079509.104448:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961593088:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079509.295966:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079509.679989:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079510.320690:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079510.704107:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079511.343968:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079511.729774:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079512.367969:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079512.751937:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079513.391944:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079513.777109:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079514.224557:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079514.224565:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079514.224569:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079514.224577:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5196 00000100:00080000:3.0:1769079514.224586:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079514.224591:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079514.224622:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961593216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079514.224643:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079514.224661:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079514.224663:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079514.224667:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5196 00000100:00080000:3.0:1769079514.224671:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079514.224675:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079514.224684:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079514.224691:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961593216:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079514.224695:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961593344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079514.224704:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079514.224707:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079514.224708:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079514.224712:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5196 00000100:00080000:3.0:1769079514.224714:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079514.224717:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079514.224721:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079514.224728:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961593472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769079514.224825:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079514.224876:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079514/real 1769079514] req@ffff895e453d4700 x1855008961593216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079514.224892:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961593216/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079514.224902:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079514.224906:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079514.224909:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961593216:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079514.224933:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961593344:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079514.224956:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079514.225554:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079514/real 1769079514] req@ffff895e453d5180 x1855008961593344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079514.225571:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961593344/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079514.225597:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079514.225601:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079514.225604:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961593344:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079514.225611:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961593472:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079514.225638:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079514.225649:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079514/real 1769079514] req@ffff895e453d6300 x1855008961593472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079514.225658:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961593472/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079514.225665:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079514.225668:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079514.225671:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961593472:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079514.418257:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079514.799829:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079515.440798:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079515.825984:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079516.473532:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079516.847863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079517.487976:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079517.872000:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079518.517404:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079518.895879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079519.343890:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079519.343901:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079519.343905:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079519.343914:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5201 00000100:00080000:3.0:1769079519.343924:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079519.343931:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079519.343966:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961593600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079519.344061:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961593600:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079519.344128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769079519.344156:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079519.344160:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079519.344162:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079519.344166:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5201 00000100:00080000:3.0:1769079519.344169:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079519.344444:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079519.344455:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079519.344469:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961593728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079519.344584:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079519/real 1769079519] req@ffff895e453d5880 x1855008961593600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079519.344600:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961593600/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079519.344621:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079519.344624:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079519.344628:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961593600:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079519.344651:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961593728:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079519.344673:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079519.344685:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079519/real 1769079519] req@ffff895e453d4380 x1855008961593728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079519.344707:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961593728/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079519.344716:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079519.344719:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079519.344722:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961593728:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769079519.345927:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079519.345934:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079519.345936:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079519.345942:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5201 00000100:00080000:3.0:1769079519.345950:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079519.345955:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079519.345963:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079519.345984:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961593856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079519.346169:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961593856:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079519.349934:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079519.350250:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079519/real 1769079519] req@ffff895e453d6680 x1855008961593856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079519.350274:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961593856/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079519.350285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079519.350294:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079519.350298:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961593856:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079519.537988:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079519.919883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079520.561264:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079520.943907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079521.583965:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079521.969331:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079522.607977:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079522.991911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079523.633152:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079524.015941:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079524.466389:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079524.466401:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079524.466406:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079524.466416:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5207 00000100:00080000:0.0:1769079524.466429:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079524.466437:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079524.466478:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961593984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079524.466644:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961593984:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079524.466711:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769079524.466742:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079524.466778:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079524.466780:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079524.466786:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5207 00000100:00080000:0.0:1769079524.466790:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079524.466794:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079524.466803:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079524.466818:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961594112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079524.469652:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079524/real 1769079524] req@ffff895e4383d180 x1855008961593984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079524.469681:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961593984/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079524.469692:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079524.469697:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079524.469701:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961593984:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079524.469736:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961594112:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079524.469808:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079524.469826:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079524/real 1769079524] req@ffff895e4383df80 x1855008961594112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079524.469838:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961594112/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079524.469847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079524.469851:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079524.469854:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961594112:192.168.204.117@tcp:38:kworker.0 00000100:00080000:0.0:1769079524.471124:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079524.471134:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079524.471137:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079524.471147:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5207 00000100:00080000:0.0:1769079524.471157:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079524.471164:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079524.471590:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079524.471633:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961594240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079524.472027:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961594240:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079524.472107:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079524.472134:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079524/real 1769079524] req@ffff895e4383ed80 x1855008961594240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079524.472153:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961594240/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079524.472162:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079524.472313:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079524.472318:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961594240:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079524.655958:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079525.040009:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079525.682330:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079526.063883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079526.703981:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079527.087902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079527.728125:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079528.113932:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079528.752474:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079529.135881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079529.584486:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079529.584498:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079529.584503:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079529.584512:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5212 00000100:00080000:1.0:1769079529.584525:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079529.584532:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079529.584571:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961594368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079529.584606:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079529.584610:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079529.584612:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079529.584616:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5212 00000100:00080000:1.0:1769079529.584620:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079529.584643:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079529.584652:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079529.584665:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961594496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079529.584674:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079529.584677:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079529.584679:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079529.584682:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5212 00000100:00080000:1.0:1769079529.584686:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079529.584690:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079529.584694:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079529.584702:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961594624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079529.585876:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961594368:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079529.585988:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961594496:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079529.586011:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961594624:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079529.776057:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079530.159958:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079530.802170:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079531.183970:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079531.824053:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079532.208985:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769079532.656070:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769079532.656112:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769079532.656130:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079532.656169:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079532.656178:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:2.0:1769079532.656290:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079529/real 1769079532] req@ffff895e4713ea00 x1855008961594368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079532.656316:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961594368/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079532.656328:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079532.656345:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079532.656349:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961594368:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079532.656357:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079529/real 1769079532] req@ffff895e4713f100 x1855008961594496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079532.656370:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961594496/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079532.656378:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079532.656381:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079532.656385:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961594496:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079532.656391:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079529/real 1769079532] req@ffff895e4713e300 x1855008961594624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079532.656402:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961594624/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079532.656409:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079532.656413:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079532.656416:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961594624:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079532.847892:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079533.231921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079533.872001:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079534.256145:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079534.705864:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079534.705876:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079534.705880:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079534.705890:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5217 00000100:00080000:0.0:1769079534.705919:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079534.705926:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079534.705962:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961594752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079534.705993:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079534.705996:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079534.705997:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079534.706000:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5217 00000100:00080000:0.0:1769079534.706003:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079534.706006:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079534.706017:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079534.706028:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961594880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079534.706038:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079534.706041:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079534.706043:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079534.706046:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5217 00000100:00080000:0.0:1769079534.706050:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079534.706054:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079534.706059:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079534.706069:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961595008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079534.706242:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961594752:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079534.707002:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079534.707059:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079534/real 1769079534] req@ffff895e4383d880 x1855008961594752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079534.707096:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961594752/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079534.707105:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079534.707109:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079534.707112:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961594752:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079534.707125:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961594880:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079534.707153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079534.707164:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079534/real 1769079534] req@ffff895e4383e680 x1855008961594880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079534.707177:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961594880/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079534.707186:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079534.707190:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079534.707193:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961594880:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079534.707199:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961595008:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079534.707217:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079534.707227:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079534/real 1769079534] req@ffff895e4383c700 x1855008961595008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079534.707238:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961595008/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079534.707256:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079534.707260:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079534.707264:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961595008:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079534.896480:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079535.281213:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079535.921096:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079536.303921:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079536.944089:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079537.328345:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079537.968332:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079538.351923:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079538.996146:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079539.376576:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079539.824017:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079539.824032:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079539.824037:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079539.824050:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5222 00000100:00080000:3.0:1769079539.824090:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079539.824098:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079539.824144:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6d80 x1855008961595136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079539.824185:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079539.824188:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079539.824191:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079539.824195:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5222 00000100:00080000:3.0:1769079539.824199:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079539.824203:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079539.824215:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079539.824230:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6680 x1855008961595264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079539.824239:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079539.824242:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079539.824245:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079539.824248:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5222 00000100:00080000:3.0:1769079539.824252:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079539.824256:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079539.824262:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079539.824271:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4380 x1855008961595392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079539.825123:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961595136:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079539.825216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079539.825267:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079539/real 1769079539] req@ffff895e453d6d80 x1855008961595136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079539.825285:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6d80 x1855008961595136/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079539.825296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079539.825301:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079539.825306:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961595136:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079539.825314:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961595264:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079539.825338:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079539.825675:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079539/real 1769079539] req@ffff895e453d6680 x1855008961595264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079539.825689:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6680 x1855008961595264/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079539.825716:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079539.825729:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079539.825733:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961595264:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079539.825740:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961595392:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079539.825813:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079539.825825:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079539/real 1769079539] req@ffff895e453d4380 x1855008961595392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079539.825837:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4380 x1855008961595392/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079539.825845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079539.825848:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079539.825852:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961595392:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079540.016095:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079540.400857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079541.044844:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079541.423933:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079542.065034:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079542.452405:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079543.089215:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079543.472135:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079544.111934:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079544.495882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079544.943841:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079544.943867:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079544.943870:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079544.943880:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5227 00000100:00080000:3.0:1769079544.943890:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079544.943897:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079544.943929:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5880 x1855008961595520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079544.944017:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961595520:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079544.944077:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769079544.944109:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079544.944113:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079544.944115:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079544.944119:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5227 00000100:00080000:3.0:1769079544.944124:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079544.944128:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079544.944137:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079544.944161:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d6300 x1855008961595648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079544.948611:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079544/real 1769079544] req@ffff895e453d5880 x1855008961595520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079544.948652:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5880 x1855008961595520/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079544.948684:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079544.948690:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079544.948694:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961595520:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079544.948729:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961595648:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079544.948801:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079544.948818:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079544/real 1769079544] req@ffff895e453d6300 x1855008961595648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079544.948830:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d6300 x1855008961595648/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079544.948836:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079544.948840:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079544.948843:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961595648:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769079544.953080:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079544.953092:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079544.953097:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079544.953106:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5227 00000100:00080000:3.0:1769079544.953118:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079544.953126:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079544.953139:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079544.953223:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5180 x1855008961595776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079544.953349:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961595776:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079544.954281:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079544.955657:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079544/real 1769079544] req@ffff895e453d5180 x1855008961595776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079544.955682:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5180 x1855008961595776/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079544.955693:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079544.955697:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079544.955701:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961595776:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079545.139146:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079545.521484:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079546.159965:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079546.543995:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079547.187453:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079547.569041:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079548.207967:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079548.592226:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079549.231957:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079549.615983:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079550.066500:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079550.066513:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079550.066517:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079550.066528:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5232 00000100:00080000:1.0:1769079550.066538:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079550.066557:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079550.066596:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961595904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079550.066641:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079550.066645:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079550.066647:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079550.066659:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5232 00000100:00080000:1.0:1769079550.066662:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079550.066666:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079550.066674:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079550.066704:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961596032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079550.066715:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079550.066718:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079550.066720:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079550.066724:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5232 00000100:00080000:1.0:1769079550.066728:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079550.066732:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079550.066737:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079550.066783:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961596160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079550.067197:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961595904:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079550.067254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079550.067289:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079550/real 1769079550] req@ffff895e4713d880 x1855008961595904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079550.067306:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961595904/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079550.067317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079550.067320:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079550.067323:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961595904:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079550.067330:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961596032:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079550.067349:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079550.067360:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079550/real 1769079550] req@ffff895e4713ed80 x1855008961596032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079550.067370:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961596032/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079550.067378:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079550.067381:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079550.067384:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961596032:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079550.067389:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961596160:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079550.067419:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079550.067429:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079550/real 1769079550] req@ffff895e4713c000 x1855008961596160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079550.067439:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961596160/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079550.067447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079550.067450:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079550.067452:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961596160:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079550.257876:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079550.639857:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079551.280297:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079551.664059:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079552.304633:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079552.689931:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079553.328345:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079553.711999:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079554.354605:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079554.735835:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079555.183881:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079555.183897:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079555.183901:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079555.183911:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5237 00000100:00080000:1.0:1769079555.183922:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079555.183929:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079555.183967:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961596288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079555.184000:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079555.184018:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079555.184020:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079555.184024:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5237 00000100:00080000:1.0:1769079555.184028:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079555.184032:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079555.184052:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079555.184064:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961596416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079555.184082:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079555.184085:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079555.184087:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079555.184090:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5237 00000100:00080000:1.0:1769079555.184093:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079555.184100:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079555.184104:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079555.184116:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961596544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079555.185402:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961596288:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079555.185485:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079555.185534:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079555/real 1769079555] req@ffff895e4713c700 x1855008961596288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079555.185554:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961596288/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079555.185564:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079555.185568:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079555.185571:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961596288:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079555.185578:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961596416:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079555.185599:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079555.185624:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079555/real 1769079555] req@ffff895e4713fb80 x1855008961596416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079555.185636:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961596416/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079555.185645:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079555.185648:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079555.185651:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961596416:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079555.186333:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961596544:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079555.186362:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079555.186375:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079555/real 1769079555] req@ffff895e4713c000 x1855008961596544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079555.186387:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961596544/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079555.186394:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079555.186397:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079555.186400:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961596544:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079555.376070:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079555.759875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079556.400026:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079556.786318:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079557.424076:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079557.808001:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079558.447956:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079558.833819:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079559.472034:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079559.856060:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079560.305088:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079560.305097:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079560.305101:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079560.305109:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5242 00000100:00080000:0.0:1769079560.305122:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079560.305128:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079560.305166:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961596672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079560.305185:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079560.305187:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079560.305189:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079560.305191:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5242 00000100:00080000:0.0:1769079560.305193:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079560.305196:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079560.305203:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079560.305211:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961596800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079560.305217:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079560.305219:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079560.305236:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079560.305238:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5242 00000100:00080000:0.0:1769079560.305241:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079560.305244:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079560.305247:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079560.305253:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961596928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079560.305922:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961596672:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079560.305987:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079560.306026:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079560/real 1769079560] req@ffff895e4383f100 x1855008961596672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079560.306040:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961596672/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079560.306047:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079560.306050:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079560.306052:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961596672:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079560.306057:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961596800:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079560.306072:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079560.306079:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079560/real 1769079560] req@ffff895e4383c380 x1855008961596800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079560.306086:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961596800/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079560.306091:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079560.306093:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079560.306095:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961596800:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079560.306098:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961596928:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079560.306107:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079560.306111:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079560/real 1769079560] req@ffff895e4383fb80 x1855008961596928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079560.306118:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961596928/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079560.306132:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079560.306134:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079560.306136:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961596928:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079560.497932:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079560.880070:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079561.522571:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079561.903880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079562.543951:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079562.928873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079563.568894:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079563.952058:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079564.593361:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079564.975945:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079565.424103:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079565.424116:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079565.424120:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079565.424131:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5247 00000100:00080000:3.0:1769079565.424142:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079565.424149:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079565.424192:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4700 x1855008961597056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079565.424228:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079565.424231:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079565.424233:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079565.424237:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5247 00000100:00080000:3.0:1769079565.424241:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079565.424245:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079565.424255:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079565.424268:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961597184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079565.424278:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079565.424281:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079565.424282:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079565.424286:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5247 00000100:00080000:3.0:1769079565.424313:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079565.424318:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079565.424341:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079565.424353:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961597312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079565.424449:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961597056:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079565.424514:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079565.424555:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079565/real 1769079565] req@ffff895e453d4700 x1855008961597056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079565.424575:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4700 x1855008961597056/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079565.424585:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079565.424588:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079565.424592:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961597056:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079565.424599:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961597184:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079565.424620:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079565.424632:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079565/real 1769079565] req@ffff895e453d7100 x1855008961597184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079565.424642:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961597184/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079565.424661:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079565.424665:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079565.424669:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961597184:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079565.424674:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961597312:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079565.424695:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079565.424704:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079565/real 1769079565] req@ffff895e453d7b80 x1855008961597312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079565.424715:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961597312/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079565.424721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079565.424723:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079565.424726:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961597312:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079565.616116:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079566.000047:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079566.640286:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079567.023879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079567.663943:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079568.047882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079568.688000:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079569.071870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079569.712064:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079570.095871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079570.543861:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079570.543872:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079570.543877:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079570.543885:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5253 00000100:00080000:3.0:1769079570.543895:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079570.543902:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079570.543938:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961597440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079570.544014:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961597440:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079570.544073:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769079570.544104:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079570.544108:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079570.544110:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079570.544114:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5253 00000100:00080000:3.0:1769079570.544118:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079570.544121:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079570.544130:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079570.544144:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961597568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079570.544975:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079570/real 1769079570] req@ffff895e453d5500 x1855008961597440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079570.544995:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961597440/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079570.545004:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079570.545023:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079570.545027:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961597440:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079570.545055:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961597568:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079570.545087:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079570.545101:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079570/real 1769079570] req@ffff895e453d4e00 x1855008961597568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079570.545113:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961597568/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079570.545121:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079570.545124:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079570.545127:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961597568:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769079570.545249:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079570.545254:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079570.545256:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079570.545260:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5253 00000100:00080000:3.0:1769079570.545270:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079570.545275:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079570.545282:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079570.545299:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961597696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079570.545350:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961597696:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079570.545375:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079570.545441:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079570/real 1769079570] req@ffff895e453d4a80 x1855008961597696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079570.545454:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961597696/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079570.545462:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079570.545465:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079570.545469:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961597696:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079570.735968:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079571.119885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079571.760879:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079572.143878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079572.785909:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079573.167865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079573.809270:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079574.191982:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079574.832110:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079575.215985:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079575.666078:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079575.666090:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079575.666094:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079575.666104:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5258 00000100:00080000:0.0:1769079575.666115:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079575.666132:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079575.666171:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961597824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079575.666258:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961597824:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079575.666322:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769079575.666354:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079575.666358:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079575.666360:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079575.666364:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5258 00000100:00080000:0.0:1769079575.666368:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079575.666388:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079575.666397:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079575.666410:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961597952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079575.668487:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079575/real 1769079575] req@ffff895e4383ce00 x1855008961597824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079575.668515:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961597824/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079575.668526:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079575.668530:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079575.668534:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961597824:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079575.668571:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961597952:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079575.668621:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079575.668637:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079575/real 1769079575] req@ffff895e4383f480 x1855008961597952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079575.668649:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961597952/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079575.668997:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079575.669002:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079575.669006:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961597952:192.168.204.117@tcp:38:kworker.0 00000100:00080000:0.0:1769079575.669100:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079575.669106:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079575.669109:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079575.669124:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5258 00000100:00080000:0.0:1769079575.669131:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079575.669136:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079575.669146:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079575.669173:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961598080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079575.669232:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961598080:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079575.669262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079575.669591:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079575/real 1769079575] req@ffff895e4383c000 x1855008961598080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079575.669622:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961598080/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079575.669629:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079575.669633:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079575.669636:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961598080:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079575.855945:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079576.239959:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079576.883692:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079577.264367:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079577.906199:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079578.289106:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079578.928036:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079579.311848:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079579.955971:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079580.335871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079580.783976:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079580.784703:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079580.784708:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079580.784718:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5263 00000100:00080000:3.0:1769079580.784730:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079580.784737:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079580.785102:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961598208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079580.785169:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961598208:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079580.785309:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769079580.785339:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079580.785343:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079580.785345:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079580.785351:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5263 00000100:00080000:3.0:1769079580.785355:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079580.785360:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079580.785369:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079580.785382:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961598336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079580.786476:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079580/real 1769079580] req@ffff895e453d7480 x1855008961598208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079580.786502:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961598208/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079580.786511:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079580.786515:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079580.786532:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961598208:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079580.786561:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961598336:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079580.786600:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079580.786614:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079580/real 1769079580] req@ffff895e453d5f80 x1855008961598336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079580.786625:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961598336/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079580.786633:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079580.786637:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079580.786640:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961598336:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769079580.786712:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079580.786723:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079580.786725:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079580.786730:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5263 00000100:00080000:3.0:1769079580.786735:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079580.786740:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079580.786763:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079580.786781:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961598464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079580.786860:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961598464:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079580.786883:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079580.786894:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079580/real 1769079580] req@ffff895e453d5c00 x1855008961598464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079580.786904:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961598464/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079580.786911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079580.786914:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079580.786917:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961598464:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079580.976293:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079581.359905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079582.002045:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079582.384047:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079583.025939:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079583.407938:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079584.048316:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079584.431870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079585.071968:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079585.455868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079585.903941:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079585.903960:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079585.903966:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079585.903978:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5268 00000100:00080000:2.0:1769079585.903989:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079585.903997:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079585.904039:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961598592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079585.904070:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079585.904074:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079585.904076:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079585.904080:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5268 00000100:00080000:2.0:1769079585.904083:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079585.904087:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079585.904095:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079585.904127:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961598720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079585.904143:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079585.904146:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079585.904148:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079585.904151:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5268 00000100:00080000:2.0:1769079585.904155:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079585.904158:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079585.904162:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079585.904171:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961598848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079585.904270:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961598592:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079585.904338:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079585.904375:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079585/real 1769079585] req@ffff895e44e2b480 x1855008961598592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079585.904392:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961598592/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079585.904401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079585.904405:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079585.904408:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961598592:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079585.904415:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961598720:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079585.904435:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079585.904447:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079585/real 1769079585] req@ffff895e44e28e00 x1855008961598720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079585.904458:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961598720/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079585.904466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079585.904469:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079585.904472:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961598720:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079585.904478:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961598848:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079585.904495:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079585.904505:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079585/real 1769079585] req@ffff895e44e29f80 x1855008961598848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079585.904529:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961598848/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079585.904536:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079585.904538:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079585.904541:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961598848:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079586.096078:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079586.479925:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079587.123778:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079587.503891:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079588.145731:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079588.531192:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079589.169193:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079589.553468:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079590.192143:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079590.575826:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079591.026071:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079591.026083:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079591.026087:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079591.026097:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5273 00000100:00080000:0.0:1769079591.026108:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079591.026115:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079591.026154:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961598976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079591.026197:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079591.026200:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079591.026202:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079591.026206:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5273 00000100:00080000:0.0:1769079591.026209:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079591.026213:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079591.026221:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079591.026250:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961599104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079591.026260:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079591.026262:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079591.026264:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079591.026268:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5273 00000100:00080000:0.0:1769079591.026272:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079591.026276:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079591.026281:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079591.026305:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961599232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079591.026483:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961598976:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079591.026540:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079591.026576:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079591/real 1769079591] req@ffff895e4383dc00 x1855008961598976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079591.026593:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961598976/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079591.026604:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079591.026608:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079591.026611:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961598976:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079591.026618:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961599104:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079591.026647:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079591.026659:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079591/real 1769079591] req@ffff895e4383ca80 x1855008961599104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079591.026671:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961599104/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079591.026680:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079591.026688:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079591.026691:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961599104:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079591.026697:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961599232:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079591.026715:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079591.026725:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079591/real 1769079591] req@ffff895e4383ea00 x1855008961599232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079591.026735:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961599232/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079591.026743:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079591.026768:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079591.026771:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961599232:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079591.216026:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079591.599915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079592.239947:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079592.626529:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079593.263957:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079593.649185:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079594.287932:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079594.671987:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079595.312050:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079595.695959:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079596.143930:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079596.143941:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079596.143945:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079596.143956:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5278 00000100:00080000:2.0:1769079596.143967:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079596.143993:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079596.144034:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961599360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079596.144106:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961599360:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769079596.144223:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079596.144228:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079596.144229:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079596.144234:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5278 00000100:00080000:2.0:1769079596.144238:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079596.144242:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079596.144252:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079596.144263:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961599488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079596.144436:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961599488:192.168.204.117@tcp:38:kworker.0 00000100:00080000:2.0:1769079596.148519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079596.148530:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079596.148534:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079596.148544:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5278 00000100:00080000:2.0:1769079596.148553:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079596.148559:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079596.148569:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079596.148605:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961599616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079596.148688:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961599616:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079596.339148:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079596.719879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079597.359975:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079597.746937:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079598.384140:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079598.767868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769079599.216101:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769079599.216151:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769079599.216169:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079599.216210:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079599.216219:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769079599.216431:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079596/real 1769079599] req@ffff895e44e29180 x1855008961599616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079599.216457:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961599616/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079599.216468:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079599.216487:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079599.216492:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961599616:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079599.216500:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079596/real 1769079599] req@ffff895e44e28a80 x1855008961599488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079599.216527:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961599488/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079599.216536:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079599.216539:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079599.216542:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961599488:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079599.216548:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079596/real 1769079599] req@ffff895e44e29500 x1855008961599360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079599.216559:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961599360/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079599.216567:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079599.216570:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079599.216573:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961599360:192.168.204.117@tcp:250:kworker.0 00010000:00010000:1.0:1769079599.412511:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079599.791987:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079600.431960:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079600.819251:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079601.263873:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079601.263892:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079601.263910:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079601.263920:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5283 00000100:00080000:2.0:1769079601.263932:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079601.263939:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079601.263976:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961599744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079601.264011:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079601.264015:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079601.264017:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079601.264021:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5283 00000100:00080000:2.0:1769079601.264025:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079601.264028:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079601.264039:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079601.264050:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961599872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079601.264060:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079601.264063:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079601.264065:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079601.264068:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5283 00000100:00080000:2.0:1769079601.264072:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079601.264075:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079601.264080:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079601.264088:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961600000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079601.264207:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961599744:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079601.264278:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079601.264314:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079601/real 1769079601] req@ffff895e44e2bb80 x1855008961599744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079601.264330:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961599744/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079601.264347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079601.264351:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079601.264356:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961599744:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079601.264363:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961599872:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079601.264384:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079601.264395:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079601/real 1769079601] req@ffff895e44e29c00 x1855008961599872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079601.264406:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961599872/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079601.264414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079601.264416:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079601.264419:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961599872:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079601.264424:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961600000:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079601.264440:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079601.264448:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079601/real 1769079601] req@ffff895e44e2b800 x1855008961600000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079601.264458:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961600000/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079601.264465:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079601.264468:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079601.264471:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961600000:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079601.458120:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079601.839872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079602.480037:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079602.864494:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079603.504053:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079603.887872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079604.531215:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079604.911917:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079605.551964:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079605.936026:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079606.385991:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079606.386003:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079606.386008:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079606.386018:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5288 00000100:00080000:0.0:1769079606.386029:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079606.386036:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079606.386078:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961600128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079606.386160:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961600128:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079606.386225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769079606.386258:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079606.386262:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079606.386264:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079606.386268:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5288 00000100:00080000:0.0:1769079606.386272:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079606.386276:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079606.386287:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079606.386300:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961600256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079606.386838:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079606/real 1769079606] req@ffff895e4383d500 x1855008961600128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079606.386857:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961600128/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079606.386867:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079606.386870:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079606.386874:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961600128:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079606.386901:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961600256:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079606.391010:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079606.391054:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079606/real 1769079606] req@ffff895e4383f800 x1855008961600256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079606.391074:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961600256/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079606.391085:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079606.391089:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079606.391093:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961600256:192.168.204.117@tcp:38:kworker.0 00000100:00080000:0.0:1769079606.393210:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079606.393220:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079606.393224:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079606.393233:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5288 00000100:00080000:0.0:1769079606.393243:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079606.393267:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079606.393279:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079606.393308:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961600384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079606.393447:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961600384:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079606.393501:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079606.393533:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079606/real 1769079606] req@ffff895e4383e300 x1855008961600384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079606.393548:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961600384/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079606.393557:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079606.393561:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079606.393564:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961600384:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079606.576078:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079606.960112:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079607.603347:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079607.983878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079608.623932:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079609.007964:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079609.648490:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079610.032117:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079610.671997:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079611.055909:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079611.503923:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079611.503933:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079611.503937:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079611.503947:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5294 00000100:00080000:0.0:1769079611.503958:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079611.503965:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079611.504004:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961600512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079611.504089:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961600512:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079611.504152:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769079611.504183:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079611.504187:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079611.504189:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079611.504193:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5294 00000100:00080000:0.0:1769079611.504197:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079611.504201:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079611.504210:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079611.504223:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961600640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079611.504359:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079611/real 1769079611] req@ffff895e4383e300 x1855008961600512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079611.504377:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961600512/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079611.504386:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079611.504390:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079611.504403:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961600512:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079611.504429:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961600640:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079611.504451:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079611.504463:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079611/real 1769079611] req@ffff895e4383f800 x1855008961600640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079611.504475:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961600640/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079611.504502:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079611.504506:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079611.504509:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961600640:192.168.204.117@tcp:38:kworker.0 00000100:00080000:0.0:1769079611.505618:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079611.505633:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079611.505635:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079611.506375:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5294 00000100:00080000:0.0:1769079611.506385:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079611.506465:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079611.506476:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079611.506503:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961600768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079611.506579:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961600768:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079611.506627:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079611.511664:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079611/real 1769079611] req@ffff895e4383d500 x1855008961600768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079611.511695:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961600768/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079611.511706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079611.512526:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079611.512535:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961600768:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079611.696583:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079612.079958:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079612.720467:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079613.106189:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079613.744023:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079614.127916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079614.767926:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079615.152103:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079615.792549:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079616.177582:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079616.625046:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079616.625058:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079616.625063:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079616.625074:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5299 00000100:00080000:2.0:1769079616.625087:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079616.625094:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079616.625140:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961600896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079616.625176:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079616.625191:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079616.625194:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079616.625198:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5299 00000100:00080000:2.0:1769079616.625203:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079616.625208:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079616.625217:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079616.625232:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961600896:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079616.625233:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961601024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079616.625248:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079616.625251:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079616.625253:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079616.625257:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5299 00000100:00080000:2.0:1769079616.625261:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079616.625265:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079616.625270:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079616.625282:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961601152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769079616.625300:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079616.625339:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079616/real 1769079616] req@ffff895e44e2b800 x1855008961600896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079616.625358:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961600896/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079616.625368:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079616.625373:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079616.625377:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961600896:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079616.625402:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961601024:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079616.625442:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079616.625455:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079616/real 1769079616] req@ffff895e44e29c00 x1855008961601024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079616.625469:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961601024/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079616.625484:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079616.625488:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079616.625492:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961601024:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079616.625498:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961601152:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079616.625519:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079616.625530:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079616/real 1769079616] req@ffff895e44e2bb80 x1855008961601152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079616.625542:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961601152/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079616.625550:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079616.625553:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079616.625556:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961601152:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079616.816406:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079617.200207:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079617.840438:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079618.223947:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079618.865075:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079619.248344:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079619.887956:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079620.272062:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079620.911975:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079621.295933:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079621.743944:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079621.743954:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079621.743958:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079621.743968:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5304 00000100:00080000:3.0:1769079621.743979:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079621.743986:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079621.744021:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961601280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079621.744087:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961601280:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079621.744159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:3.0:1769079621.744330:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079621.744357:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079621.744359:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079621.744364:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5304 00000100:00080000:3.0:1769079621.744368:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079621.744372:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079621.744380:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079621.744392:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961601408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079621.744442:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079621/real 1769079621] req@ffff895e453d7800 x1855008961601280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079621.744458:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961601280/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079621.744476:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079621.744479:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079621.744483:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961601280:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079621.744505:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961601408:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079621.744524:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079621.744534:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079621/real 1769079621] req@ffff895e453d5c00 x1855008961601408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079621.744545:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961601408/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079621.744552:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079621.744555:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079621.744558:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961601408:192.168.204.117@tcp:38:kworker.0 00000100:00080000:3.0:1769079621.744635:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079621.744639:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079621.744641:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079621.744645:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5304 00000100:00080000:3.0:1769079621.744649:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079621.744652:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079621.744657:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079621.744668:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961601536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079621.744702:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961601536:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079621.744721:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079621.744877:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079621/real 1769079621] req@ffff895e453d5f80 x1855008961601536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079621.744889:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961601536/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079621.744896:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079621.744899:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079621.744902:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961601536:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079621.936156:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079622.319864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079622.960113:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079623.343868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079623.986989:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079624.367815:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079625.008333:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079625.391859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079626.031953:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079626.416063:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079626.863957:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079626.863970:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079626.863975:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079626.863986:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5309 00000100:00080000:0.0:1769079626.863998:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079626.864006:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079626.864047:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961601664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079626.864082:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079626.864086:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079626.864088:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079626.864092:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5309 00000100:00080000:0.0:1769079626.864111:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079626.864116:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079626.864126:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079626.864140:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961601792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079626.864149:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079626.864152:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079626.864154:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079626.864170:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5309 00000100:00080000:0.0:1769079626.864174:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079626.864177:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079626.864182:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079626.864192:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961601920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079626.864232:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961601664:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079626.864308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079626.864347:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079626/real 1769079626] req@ffff895e4383ea00 x1855008961601664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079626.864366:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961601664/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079626.864376:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079626.864380:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079626.864384:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961601664:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079626.864392:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961601792:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079626.864413:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079626.864425:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079626/real 1769079626] req@ffff895e4383ca80 x1855008961601792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079626.864436:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961601792/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079626.864444:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079626.864447:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079626.864450:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961601792:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079626.864456:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961601920:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079626.864475:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079626.864485:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079626/real 1769079626] req@ffff895e4383dc00 x1855008961601920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079626.864503:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961601920/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079626.864511:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079626.864514:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079626.864517:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961601920:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079627.056010:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079627.439888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079628.080103:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079628.463869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079629.104554:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079629.487869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079630.128068:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079630.511861:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079631.152147:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079631.537617:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079631.990898:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079631.990913:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079631.990919:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079631.990937:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5314 00000100:00080000:0.0:1769079631.990952:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079631.990961:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079631.991013:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961602048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079631.991076:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079631.991080:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079631.991082:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079631.991087:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5314 00000100:00080000:0.0:1769079631.991091:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079631.991096:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079631.991107:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079631.991257:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961602048:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079631.991316:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961602176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079631.991334:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079631.991338:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079631.991341:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1769079631.991349:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:0.0:1769079631.991354:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5314 00000100:00080000:0.0:1769079631.991363:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079631.991368:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079631.991374:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079631.991392:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961602304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079631.991411:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079631/real 1769079631] req@ffff895e4383c000 x1855008961602048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079631.991434:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961602048/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079631.991447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079631.991452:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079631.991457:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961602048:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079631.991487:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961602176:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079631.991512:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079631.991525:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079631/real 1769079631] req@ffff895e4383f480 x1855008961602176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079631.991537:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961602176/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079631.991564:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079631.991568:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079631.991571:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961602176:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079631.991577:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961602304:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079631.991600:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079631.991610:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079631/real 1769079631] req@ffff895e4383ce00 x1855008961602304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079631.991621:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961602304/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079631.991629:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079631.991632:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079631.991635:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961602304:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079632.182315:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079632.560352:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079633.204355:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079633.584355:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079634.223952:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079634.607892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079635.248325:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079635.632264:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079636.271933:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079636.655904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079637.104324:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079637.104336:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079637.104340:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079637.104351:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5319 00000100:00080000:2.0:1769079637.104364:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079637.104372:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079637.104410:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961602432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079637.104444:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079637.104467:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079637.104470:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079637.104475:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5319 00000100:00080000:2.0:1769079637.104479:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079637.104483:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079637.104492:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079637.104506:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961602560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079637.104516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079637.104519:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079637.104521:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079637.104524:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5319 00000100:00080000:2.0:1769079637.104528:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079637.104531:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079637.104535:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079637.104546:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961602688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079637.104611:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961602432:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079637.104674:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079637.104732:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079637/real 1769079637] req@ffff895e44e29500 x1855008961602432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079637.104777:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961602432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079637.104800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079637.104804:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079637.104809:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961602432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079637.104817:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961602560:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079637.104842:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079637.104854:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079637/real 1769079637] req@ffff895e44e28a80 x1855008961602560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079637.104866:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961602560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079637.104874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079637.104877:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079637.104880:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961602560:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079637.104886:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961602688:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079637.104902:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079637.104912:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079637/real 1769079637] req@ffff895e44e29180 x1855008961602688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079637.104923:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961602688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079637.104930:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079637.104933:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079637.104936:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961602688:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079637.295970:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079637.681854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079638.321524:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079638.703908:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079639.343942:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079639.727877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079640.368156:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079640.751919:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079641.393042:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079641.776464:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079642.225244:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079642.225254:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079642.225258:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079642.225267:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5324 00000100:00080000:1.0:1769079642.225278:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079642.225284:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079642.225333:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961602816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079642.225378:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079642.225383:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079642.225384:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079642.225388:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5324 00000100:00080000:1.0:1769079642.225392:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079642.225396:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079642.225404:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079642.225417:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961602944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079642.225426:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079642.225428:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079642.225430:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079642.225433:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5324 00000100:00080000:1.0:1769079642.225436:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079642.225439:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079642.225443:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079642.225452:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961603072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079642.225475:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961602816:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079642.225534:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079642.225613:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079642/real 1769079642] req@ffff895e4713ed80 x1855008961602816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079642.225627:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961602816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079642.225636:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079642.225638:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079642.225642:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961602816:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079642.225649:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961602944:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079642.225672:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079642.225683:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079642/real 1769079642] req@ffff895e4713d880 x1855008961602944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079642.225694:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961602944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079642.225701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079642.225704:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079642.225707:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961602944:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079642.225712:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961603072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079642.225730:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079642.225738:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079642/real 1769079642] req@ffff895e4713e300 x1855008961603072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079642.225781:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961603072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079642.225789:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079642.225792:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079642.225795:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961603072:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079642.419430:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079642.801890:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079643.440210:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079643.824011:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079644.469273:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079644.847948:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079645.489485:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079645.872371:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079646.511953:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079646.895901:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079647.344242:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079647.344252:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079647.344256:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079647.344275:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5329 00000100:00080000:2.0:1769079647.344414:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079647.344423:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079647.344459:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961603200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079647.344500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079647.344504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079647.344505:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079647.344509:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5329 00000100:00080000:2.0:1769079647.344513:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079647.344518:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079647.344526:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079647.344538:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961603328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079647.344547:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961603200:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769079647.344549:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079647.344556:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079647.344557:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079647.344561:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5329 00000100:00080000:2.0:1769079647.344564:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079647.344568:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079647.344572:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079647.344581:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961603456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769079647.344613:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079647.344658:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079647/real 1769079647] req@ffff895e44e29f80 x1855008961603200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079647.344676:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961603200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079647.344686:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079647.344689:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079647.344693:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961603200:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079647.344717:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961603328:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079647.344738:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079647.344774:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079647/real 1769079647] req@ffff895e44e28e00 x1855008961603328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079647.344787:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961603328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079647.344800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079647.344804:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079647.344807:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961603328:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079647.344813:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961603456:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079647.344848:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079647.344860:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079647/real 1769079647] req@ffff895e44e2b480 x1855008961603456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079647.344872:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961603456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079647.344891:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079647.344894:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079647.344898:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961603456:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079647.536183:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079647.920025:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079648.559985:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079648.944018:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079649.584070:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079649.968134:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079650.608092:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079650.992256:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079651.632008:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079652.019821:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079652.466310:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079652.466325:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079652.466330:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079652.466343:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5335 00000100:00080000:0.0:1769079652.466358:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079652.466366:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079652.466411:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961603584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079652.466447:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079652.466450:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079652.466452:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079652.466456:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5335 00000100:00080000:0.0:1769079652.466459:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079652.466462:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079652.466474:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079652.466483:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961603712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079652.466501:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079652.466504:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079652.466506:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079652.466528:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5335 00000100:00080000:0.0:1769079652.466532:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079652.466535:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079652.466540:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079652.466549:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961603840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079652.466651:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961603584:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079652.466732:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079652.466811:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079652/real 1769079652] req@ffff895e4383fb80 x1855008961603584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079652.466846:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961603584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079652.466866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079652.466870:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079652.466875:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961603584:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079652.466882:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961603712:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079652.466904:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079652.466916:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079652/real 1769079652] req@ffff895e4383c380 x1855008961603712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079652.466927:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961603712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079652.466935:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079652.466938:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079652.466941:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961603712:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079652.466946:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961603840:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079652.466963:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079652.466972:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079652/real 1769079652] req@ffff895e4383f100 x1855008961603840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079652.466983:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961603840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079652.466989:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079652.466992:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079652.466995:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961603840:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079652.657441:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079653.041994:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079653.681183:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079654.063950:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079654.705972:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079655.087916:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079655.732271:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079656.112972:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079656.752187:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079657.136417:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079657.584631:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079657.584645:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079657.584652:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079657.584665:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5340 00000100:00080000:1.0:1769079657.584680:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079657.584689:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079657.584774:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961603968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079657.584812:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079657.584816:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079657.584818:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079657.584822:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5340 00000100:00080000:1.0:1769079657.584827:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079657.584831:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079657.584843:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079657.584857:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961604096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079657.584867:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079657.584871:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079657.584892:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079657.584896:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5340 00000100:00080000:1.0:1769079657.584901:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079657.584905:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079657.584911:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079657.584924:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961604224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079657.585044:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961603968:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079657.585151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079657.585200:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079657/real 1769079657] req@ffff895e4713f100 x1855008961603968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079657.585221:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961603968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079657.585232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079657.585237:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079657.585242:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961603968:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079657.587052:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961604096:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079657.587594:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079657.587632:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079657/real 1769079657] req@ffff895e4713ea00 x1855008961604096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079657.587652:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961604096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079657.587664:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079657.587669:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079657.587674:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961604096:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079657.587705:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961604224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079657.587731:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079657.587785:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079657/real 1769079657] req@ffff895e4713ce00 x1855008961604224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079657.587798:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961604224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079657.587806:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079657.587810:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079657.587814:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961604224:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079657.779568:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079658.159863:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079658.799966:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079659.184247:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079659.824224:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079660.209019:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079660.848302:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079661.234876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079661.872128:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079662.258153:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079662.705139:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079662.705166:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079662.705170:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079662.705180:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5345 00000100:00080000:2.0:1769079662.705190:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079662.705197:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079662.705234:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961604352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079662.705266:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079662.705269:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079662.705271:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079662.705275:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5345 00000100:00080000:2.0:1769079662.705278:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079662.705281:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079662.705290:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079662.705299:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961604480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079662.705308:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079662.705311:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079662.705313:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079662.705316:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5345 00000100:00080000:2.0:1769079662.705318:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079662.705321:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079662.705324:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079662.705331:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961604608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079662.705414:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961604352:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079662.705507:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961604480:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079662.705529:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961604608:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079662.895980:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079663.280212:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079663.919950:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079664.306437:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079664.943927:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079665.328597:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769079665.777037:0:2391:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769079665.777082:0:2391:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769079665.777100:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079665.777142:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079665.777151:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769079665.777347:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079662/real 1769079665] req@ffff895e44e2b100 x1855008961604352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079665.777372:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961604352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079665.777396:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079665.777402:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079665.777406:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961604352:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079665.777414:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079662/real 1769079665] req@ffff895e44e2a300 x1855008961604480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079665.777427:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961604480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079665.777434:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079665.777438:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079665.777441:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961604480:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079665.780149:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079662/real 1769079665] req@ffff895e44e2aa00 x1855008961604608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079665.780192:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961604608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079665.780203:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079665.780208:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079665.780212:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961604608:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079665.968492:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079666.351894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079666.992198:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079667.376942:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079667.823859:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079667.823870:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079667.823874:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079667.823883:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5350 00000100:00080000:2.0:1769079667.823893:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079667.823901:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079667.823936:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961604736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079667.823972:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079667.823975:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079667.823977:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079667.823981:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5350 00000100:00080000:2.0:1769079667.823984:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079667.823988:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079667.823997:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079667.824007:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961604864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079667.824016:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079667.824019:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079667.824021:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079667.824024:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5350 00000100:00080000:2.0:1769079667.824042:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079667.824046:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079667.824051:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079667.824061:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961604992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079667.824168:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961604736:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079667.824230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079667.824262:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079667/real 1769079667] req@ffff895e44e28000 x1855008961604736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079667.824278:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961604736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079667.824287:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079667.824290:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079667.824294:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961604736:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079667.824300:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961604864:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079667.824318:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079667.824329:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079667/real 1769079667] req@ffff895e44e28380 x1855008961604864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079667.824340:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961604864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079667.824347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079667.824355:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079667.824359:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961604864:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079667.824365:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961604992:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079667.824386:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079667.824396:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079667/real 1769079667] req@ffff895e44e29880 x1855008961604992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079667.824406:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961604992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079667.824412:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079667.824415:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079667.824418:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961604992:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079668.016031:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079668.400354:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079669.040212:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079669.424073:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079670.068781:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079670.447874:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079671.088002:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079671.474474:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079672.112130:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079672.496658:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079672.943870:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079672.943882:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079672.943886:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079672.943897:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5355 00000100:00080000:2.0:1769079672.943908:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079672.943915:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079672.943954:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961605120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079672.943988:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079672.943992:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079672.943994:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079672.943997:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5355 00000100:00080000:2.0:1769079672.944001:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079672.944005:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079672.944014:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079672.944027:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961605248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079672.944056:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079672.944059:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079672.944061:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079672.944065:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5355 00000100:00080000:2.0:1769079672.944069:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079672.944073:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079672.944077:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079672.944088:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961605376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079672.944190:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961605120:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079672.944254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079672.944298:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079672/real 1769079672] req@ffff895e44e29880 x1855008961605120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079672.944316:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961605120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079672.944324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079672.944328:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079672.944331:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961605120:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079672.944337:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961605248:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079672.944368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079672.944379:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079672/real 1769079672] req@ffff895e44e28380 x1855008961605248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079672.944391:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961605248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079672.944398:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079672.944401:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079672.944404:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961605248:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079672.944409:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961605376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079672.944428:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079672.944437:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079672/real 1769079672] req@ffff895e44e28000 x1855008961605376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079672.944448:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961605376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079672.944455:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079672.944458:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079672.944462:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961605376:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079673.140110:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079673.519871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079674.160538:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079674.544394:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079675.184035:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079675.567894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079676.209344:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079676.592172:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079677.234739:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079677.615872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079678.064227:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079678.064237:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079678.064241:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079678.064251:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5360 00000100:00080000:3.0:1769079678.064262:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079678.064270:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079678.064314:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961605504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079678.064343:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079678.064346:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079678.064348:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079678.064351:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5360 00000100:00080000:3.0:1769079678.064354:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079678.064358:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079678.064367:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079678.064378:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961605632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079678.064387:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079678.064412:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079678.064414:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079678.064418:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5360 00000100:00080000:3.0:1769079678.064422:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079678.064425:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079678.064430:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079678.064440:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961605760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079678.064688:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961605504:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079678.064802:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079678.064855:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079678/real 1769079678] req@ffff895e453d7480 x1855008961605504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079678.064888:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961605504/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079678.064897:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079678.064902:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079678.064905:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961605504:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079678.064912:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961605632:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079678.064935:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079678.064946:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079678/real 1769079678] req@ffff895e453d4a80 x1855008961605632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079678.064957:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961605632/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079678.064964:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079678.064967:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079678.064970:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961605632:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079678.064976:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961605760:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079678.064993:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079678.065003:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079678/real 1769079678] req@ffff895e453d4e00 x1855008961605760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079678.065013:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961605760/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079678.065029:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079678.065033:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079678.065036:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961605760:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079678.256167:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079678.639868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079679.280193:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079679.664051:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079680.304727:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079680.687871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079681.327934:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079681.711872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079682.351971:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079682.735880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079683.183937:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079683.183946:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079683.183952:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079683.183962:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5365 00000100:00080000:0.0:1769079683.183972:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079683.183978:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079683.184027:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961605888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079683.184059:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079683.184062:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079683.184063:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079683.184067:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5365 00000100:00080000:0.0:1769079683.184070:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079683.184074:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079683.184090:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079683.184101:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961606016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079683.184110:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079683.184113:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079683.184128:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079683.184132:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5365 00000100:00080000:0.0:1769079683.184136:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079683.184140:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079683.184145:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079683.184157:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961606144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079683.184242:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961605888:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079683.184302:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079683.184348:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079683/real 1769079683] req@ffff895e4383c700 x1855008961605888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079683.184366:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961605888/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079683.184375:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079683.184379:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079683.184383:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961605888:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079683.184390:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961606016:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079683.184410:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079683.184430:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079683/real 1769079683] req@ffff895e4383e680 x1855008961606016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079683.184442:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961606016/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079683.184462:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079683.184465:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079683.184468:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961606016:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079683.184474:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961606144:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079683.184494:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079683.184504:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079683/real 1769079683] req@ffff895e4383d880 x1855008961606144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079683.184514:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961606144/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079683.184521:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079683.184524:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079683.184539:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961606144:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079683.385439:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079683.759878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079684.400154:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079684.783883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079685.424286:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079685.807962:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079686.448968:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079686.831873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079687.477116:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079687.856795:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079688.304130:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079688.304139:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079688.304144:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079688.304153:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5370 00000100:00080000:2.0:1769079688.304163:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079688.304170:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079688.304206:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961606272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079688.304243:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079688.304247:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079688.304263:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079688.304267:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5370 00000100:00080000:2.0:1769079688.304270:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079688.304274:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079688.304283:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079688.304294:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961606400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079688.304304:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079688.304306:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079688.304308:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079688.304311:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5370 00000100:00080000:2.0:1769079688.304313:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079688.304317:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079688.304321:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079688.304327:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961606528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079688.304401:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961606272:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079688.304456:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079688.304491:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079688/real 1769079688] req@ffff895e44e2aa00 x1855008961606272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079688.304507:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961606272/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079688.304516:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079688.304520:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079688.304524:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961606272:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079688.304530:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961606400:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079688.304545:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079688.304564:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079688/real 1769079688] req@ffff895e44e2a300 x1855008961606400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079688.304575:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961606400/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079688.304583:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079688.304586:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079688.304589:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961606400:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079688.304594:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961606528:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079688.304608:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079688.304615:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079688/real 1769079688] req@ffff895e44e2b100 x1855008961606528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079688.304625:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961606528/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079688.304653:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079688.304656:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079688.304659:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961606528:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079688.497954:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079688.879892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079689.527377:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079689.904354:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079690.543923:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079690.932673:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079691.568619:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079691.951866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079692.592088:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079692.975855:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079693.427265:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079693.427274:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079693.427278:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079693.427286:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5375 00000100:00080000:2.0:1769079693.427296:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079693.427302:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079693.427334:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961606656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079693.427377:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079693.427380:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079693.427382:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079693.427386:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5375 00000100:00080000:2.0:1769079693.427389:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079693.427393:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079693.427400:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079693.427411:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961606784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079693.427420:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079693.427422:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079693.427424:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079693.427427:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5375 00000100:00080000:2.0:1769079693.427431:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079693.427434:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079693.427439:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079693.427449:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961606912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079693.429419:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961606656:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079693.429510:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079693.429553:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079693/real 1769079693] req@ffff895e44e2b480 x1855008961606656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079693.429572:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961606656/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079693.429581:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079693.429586:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079693.429589:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961606656:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079693.429596:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961606784:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079693.429616:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079693.429627:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079693/real 1769079693] req@ffff895e44e28e00 x1855008961606784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079693.429638:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961606784/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079693.429658:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079693.429661:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079693.429664:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961606784:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079693.429670:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961606912:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079693.429691:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079693.429701:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079693/real 1769079693] req@ffff895e44e29f80 x1855008961606912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079693.429710:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961606912/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079693.429717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079693.429720:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079693.429722:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961606912:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079693.616952:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079693.999867:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079694.643910:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079695.026190:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079695.665067:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079696.051179:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079696.688107:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079697.071885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079697.712194:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079698.097464:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079698.543931:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079698.543941:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079698.543946:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079698.543957:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5381 00000100:00080000:1.0:1769079698.543968:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079698.543976:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079698.544016:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961607040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079698.544056:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079698.544060:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079698.544062:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079698.544067:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5381 00000100:00080000:1.0:1769079698.544071:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079698.544075:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079698.544084:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079698.544098:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961607168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079698.544133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079698.544137:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079698.544138:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079698.544142:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5381 00000100:00080000:1.0:1769079698.544146:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079698.544150:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079698.544155:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079698.544167:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961607296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079698.544546:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961607040:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079698.544615:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079698.544728:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079698/real 1769079698] req@ffff895e4713d500 x1855008961607040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079698.544800:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961607040/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079698.544819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079698.544823:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079698.544828:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961607040:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079698.544835:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961607168:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079698.544864:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079698.544878:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079698/real 1769079698] req@ffff895e4713f480 x1855008961607168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079698.544890:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961607168/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079698.544899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079698.544901:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079698.544904:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961607168:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079698.544911:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961607296:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079698.544927:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079698.544936:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079698/real 1769079698] req@ffff895e4713ca80 x1855008961607296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079698.544945:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961607296/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079698.544952:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079698.544954:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079698.544957:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961607296:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079698.736038:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079699.119873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079699.760065:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079700.144183:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079700.786567:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079701.168109:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079701.807981:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079702.192155:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079702.833294:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079703.216080:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079703.664097:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079703.664119:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079703.664124:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079703.664135:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5386 00000100:00080000:1.0:1769079703.664147:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079703.664154:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079703.664204:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961607424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079703.664239:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079703.664243:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079703.664244:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079703.664264:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5386 00000100:00080000:1.0:1769079703.664268:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079703.664274:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079703.664283:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079703.664294:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961607552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079703.664304:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079703.664307:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079703.664309:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079703.664312:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5386 00000100:00080000:1.0:1769079703.664315:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079703.664318:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079703.664322:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079703.664330:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961607680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079703.664920:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961607424:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079703.665082:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079703.665133:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079703/real 1769079703] req@ffff895e4713f800 x1855008961607424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079703.665151:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961607424/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079703.665160:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079703.665164:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079703.665167:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961607424:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079703.665174:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961607552:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079703.665195:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079703.665207:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079703/real 1769079703] req@ffff895e4713dc00 x1855008961607552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079703.665218:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961607552/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079703.665226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079703.665229:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079703.665233:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961607552:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079703.665251:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961607680:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079703.665272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079703.665283:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079703/real 1769079703] req@ffff895e4713df80 x1855008961607680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079703.665293:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961607680/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079703.665300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079703.665304:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079703.665307:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961607680:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079703.856090:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079704.239962:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079704.883859:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079705.264243:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079705.905536:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079706.288199:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079706.929889:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079707.311982:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079707.956135:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079708.335875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079708.785718:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079708.785829:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079708.785836:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079708.785850:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5391 00000100:00080000:0.0:1769079708.785864:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079708.785872:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079708.785999:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961607808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079708.786149:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961607808:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079708.786230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079708.786297:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079708/real 1769079708] req@ffff895e4383ed80 x1855008961607808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079708.786317:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961607808/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079708.786327:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079708.786332:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079708.786336:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961607808:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769079708.786422:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079708.786427:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079708.786429:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079708.786434:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5391 00000100:00080000:0.0:1769079708.786439:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079708.786443:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079708.786455:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079708.786470:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961607936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079708.786503:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079708.786507:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079708.786509:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079708.786514:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5391 00000100:00080000:0.0:1769079708.786518:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079708.786522:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079708.786527:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079708.786538:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961608064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079708.786649:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961607936:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079708.786675:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079708.786689:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079708/real 1769079708] req@ffff895e4383df80 x1855008961607936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079708.786701:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961607936/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079708.786709:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079708.786712:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079708.786715:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961607936:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079708.786720:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961608064:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079708.788113:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079708.788159:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079708/real 1769079708] req@ffff895e4383d180 x1855008961608064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079708.788179:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961608064/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079708.788188:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079708.788192:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079708.788196:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961608064:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079708.977228:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079709.360050:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079710.001608:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079710.385769:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079711.023953:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079711.407882:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079712.051632:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079712.431900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079713.071955:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079713.455961:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079713.906104:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079713.906116:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079713.906120:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079713.906147:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5396 00000100:00080000:1.0:1769079713.906160:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079713.906167:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079713.906368:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961608192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079713.906460:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961608192:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079713.906521:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:1.0:1769079713.906553:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079713.906556:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079713.906558:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079713.906563:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5396 00000100:00080000:1.0:1769079713.906567:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079713.906580:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079713.906590:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079713.906606:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961608320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079713.906970:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079713/real 1769079713] req@ffff895e4713d180 x1855008961608192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079713.907164:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961608192/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079713.907174:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079713.907178:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079713.907181:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961608192:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079713.907413:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961608320:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079713.907451:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079713.907467:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079713/real 1769079713] req@ffff895e4713c380 x1855008961608320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079713.907479:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961608320/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079713.907488:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079713.907492:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079713.907495:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961608320:192.168.204.117@tcp:38:kworker.0 00000100:00080000:1.0:1769079713.907830:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079713.907835:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079713.907838:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079713.907842:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5396 00000100:00080000:1.0:1769079713.907847:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079713.907852:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079713.907859:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079713.907877:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961608448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079713.907943:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961608448:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079713.907970:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079713.914557:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079713/real 1769079713] req@ffff895e4713c380 x1855008961608448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079713.914774:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961608448/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079713.914786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079713.914801:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079713.914804:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961608448:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079714.096065:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079714.484702:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079715.120075:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079715.504284:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079716.149348:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079716.527907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079717.168513:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079717.553875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079718.192502:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079718.576743:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079719.023884:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079719.023895:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079719.023900:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079719.023910:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5401 00000100:00080000:1.0:1769079719.023920:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079719.023926:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079719.023962:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961608576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079719.024034:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961608576:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079719.024093:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:1.0:1769079719.024136:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079719.024139:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079719.024141:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079719.024145:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5401 00000100:00080000:1.0:1769079719.024150:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079719.024153:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079719.024162:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079719.025299:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961608704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079719.026631:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079719/real 1769079719] req@ffff895e4713d180 x1855008961608576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079719.026661:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961608576/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079719.026670:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079719.026674:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079719.026678:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961608576:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079719.026720:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961608704:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079719.026798:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079719.026817:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079719/real 1769079719] req@ffff895e4713df80 x1855008961608704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079719.026829:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961608704/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079719.026836:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079719.026840:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079719.026843:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961608704:192.168.204.117@tcp:38:kworker.0 00000100:00080000:1.0:1769079719.028486:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079719.028495:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079719.028498:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079719.028506:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5401 00000100:00080000:1.0:1769079719.028515:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079719.028521:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079719.028534:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079719.028565:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961608832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079719.028672:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961608832:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079719.028738:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079719.028785:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079719/real 1769079719] req@ffff895e4713dc00 x1855008961608832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079719.028820:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961608832/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079719.028829:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079719.028833:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079719.028836:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961608832:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079719.216115:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079719.600316:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079720.240738:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079720.623910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079721.271820:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079721.647875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079722.294161:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079722.672242:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079723.312775:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079723.696169:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079724.144484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079724.144500:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079724.144506:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079724.144516:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5406 00000100:00080000:2.0:1769079724.144530:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079724.144538:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079724.144576:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961608960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079724.144611:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079724.144626:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079724.144629:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079724.144633:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5406 00000100:00080000:2.0:1769079724.144636:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079724.144649:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079724.144658:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079724.144671:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961609088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079724.144698:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079724.144701:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079724.144715:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079724.144719:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5406 00000100:00080000:2.0:1769079724.144724:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079724.144728:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079724.144734:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079724.144774:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961609216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079724.146128:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961608960:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079724.146212:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079724.146253:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079724/real 1769079724] req@ffff895e44e29180 x1855008961608960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079724.146272:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961608960/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079724.146282:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079724.146286:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079724.146290:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961608960:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079724.146297:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961609088:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079724.146320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079724.146345:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079724/real 1769079724] req@ffff895e44e28a80 x1855008961609088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079724.146356:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961609088/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079724.146364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079724.146366:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079724.146369:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961609088:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079724.146376:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961609216:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079724.146396:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079724.146406:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079724/real 1769079724] req@ffff895e44e29500 x1855008961609216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079724.146416:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961609216/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079724.146423:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079724.146425:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079724.146428:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961609216:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079724.339258:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079724.719915:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079725.363851:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079725.743900:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079726.384509:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079726.770357:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079727.408005:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079727.791902:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079728.434139:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079728.815963:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079729.263913:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079729.263925:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079729.263929:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079729.263950:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5411 00000100:00080000:1.0:1769079729.263982:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079729.263990:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079729.264036:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961609344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079729.264067:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079729.264071:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079729.264073:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079729.264077:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5411 00000100:00080000:1.0:1769079729.264080:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079729.264084:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079729.264092:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079729.264105:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961609472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079729.264114:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079729.264118:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079729.264119:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079729.264136:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5411 00000100:00080000:1.0:1769079729.264140:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079729.264144:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079729.264148:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079729.264158:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961609600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079729.264330:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961609344:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079729.264437:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961609472:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079729.264464:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961609600:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079729.456555:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079729.844267:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079730.481901:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079730.863869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079731.504107:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079731.888197:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769079732.336091:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769079732.336136:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769079732.336153:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079732.336192:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079732.336201:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:0.0:1769079732.336437:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079729/real 1769079732] req@ffff895e4713f800 x1855008961609344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079732.336464:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961609344/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079732.336474:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079732.336487:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079732.336491:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961609344:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079732.336499:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079729/real 1769079732] req@ffff895e4713ca80 x1855008961609472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079732.336511:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961609472/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079732.336518:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079732.336521:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079732.336524:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961609472:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079732.336530:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079729/real 1769079732] req@ffff895e4713f480 x1855008961609600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079732.336541:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961609600/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079732.336547:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079732.336551:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079732.336570:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961609600:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079732.529811:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079732.913783:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079733.552178:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079733.935865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079734.392051:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079734.392062:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079734.393258:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079734.393270:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5416 00000100:00080000:1.0:1769079734.393281:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079734.393288:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079734.393326:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961609728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079734.393427:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079734.393432:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079734.393434:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079734.393438:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5416 00000100:00080000:1.0:1769079734.393452:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079734.393457:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079734.393466:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079734.393480:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961609856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079734.393490:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079734.393512:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079734.393514:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079734.393518:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5416 00000100:00080000:1.0:1769079734.393522:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079734.393526:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079734.393531:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079734.394194:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961609984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079734.394316:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961609728:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079734.394375:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079734.394406:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079734/real 1769079734] req@ffff895e4713d500 x1855008961609728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079734.394418:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961609728/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079734.394425:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079734.394428:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079734.394431:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961609728:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079734.394435:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961609856:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079734.394463:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079734.394475:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079734/real 1769079734] req@ffff895e4713ce00 x1855008961609856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079734.394491:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961609856/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079734.394499:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079734.394502:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079734.394505:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961609856:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079734.394511:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961609984:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079734.394529:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079734.394556:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079734/real 1769079734] req@ffff895e4713ea00 x1855008961609984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079734.394569:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961609984/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079734.394577:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079734.394580:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079734.394583:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961609984:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079734.578825:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079734.961510:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079735.600217:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079735.986736:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079736.624268:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079737.007878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079737.648221:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079738.032055:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079738.672025:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079739.055885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079739.505738:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079739.505792:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079739.505799:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079739.505812:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5422 00000100:00080000:0.0:1769079739.505827:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079739.505835:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079739.505905:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961610112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079739.505943:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079739.505947:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079739.505949:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079739.505953:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5422 00000100:00080000:0.0:1769079739.505957:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079739.505962:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079739.505973:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079739.506006:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961610240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079739.506018:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079739.506022:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079739.506024:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079739.506028:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5422 00000100:00080000:0.0:1769079739.506032:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079739.506036:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079739.506041:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079739.506051:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961610368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079739.506380:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961610112:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079739.506464:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079739.506513:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079739/real 1769079739] req@ffff895e4383d180 x1855008961610112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079739.506532:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961610112/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079739.506544:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079739.506548:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079739.506552:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961610112:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079739.506567:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961610240:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079739.506592:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079739.506606:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079739/real 1769079739] req@ffff895e4383df80 x1855008961610240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079739.506619:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961610240/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079739.506628:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079739.506631:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079739.506634:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961610240:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079739.506640:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961610368:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079739.506659:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079739.506670:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079739/real 1769079739] req@ffff895e4383ed80 x1855008961610368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079739.506682:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961610368/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079739.506691:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079739.506694:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079739.506698:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961610368:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079739.696775:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079740.079946:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079740.720324:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079741.104190:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079741.744379:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079742.127992:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079742.768518:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079743.151885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079743.792372:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079744.175883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079744.624812:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079744.624822:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079744.624826:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079744.624834:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5427 00000100:00080000:0.0:1769079744.624852:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079744.624860:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079744.624898:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961610496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079744.624931:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079744.624935:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079744.624936:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079744.624957:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5427 00000100:00080000:0.0:1769079744.624962:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079744.624966:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079744.624976:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079744.624989:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961610624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079744.624998:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079744.625001:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079744.625003:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079744.625006:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5427 00000100:00080000:0.0:1769079744.625009:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079744.625012:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079744.625017:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079744.625026:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961610752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079744.625373:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961610496:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079744.625442:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079744.625482:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079744/real 1769079744] req@ffff895e4383d880 x1855008961610496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079744.625502:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961610496/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079744.625511:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079744.625515:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079744.625519:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961610496:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079744.625526:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961610624:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079744.625548:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079744.625559:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079744/real 1769079744] req@ffff895e4383e680 x1855008961610624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079744.625572:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961610624/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079744.625580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079744.625583:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079744.625586:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961610624:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079744.625591:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961610752:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079744.625610:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079744.625620:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079744/real 1769079744] req@ffff895e4383c700 x1855008961610752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079744.625643:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961610752/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079744.625651:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079744.625654:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079744.625657:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961610752:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079744.824481:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079745.199965:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079745.840724:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079746.223870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079746.863962:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079747.248078:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079747.891626:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079748.272011:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079748.913010:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079749.295924:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079749.743878:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079749.743889:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079749.743893:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079749.743903:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5432 00000100:00080000:0.0:1769079749.743915:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079749.743923:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079749.743959:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961610880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079749.744013:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079749.744026:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079749.744028:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079749.744033:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5432 00000100:00080000:0.0:1769079749.744037:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079749.744041:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079749.744050:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079749.744075:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961611008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079749.744085:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079749.744088:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079749.744090:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079749.744093:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5432 00000100:00080000:0.0:1769079749.744096:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079749.744100:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079749.744104:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079749.744115:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961611136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079749.744275:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961610880:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079749.744336:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079749.744372:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079749/real 1769079749] req@ffff895e4383f100 x1855008961610880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079749.744388:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961610880/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079749.744398:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079749.744402:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079749.744406:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961610880:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079749.744413:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961611008:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079749.744432:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079749.744442:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079749/real 1769079749] req@ffff895e4383c380 x1855008961611008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079749.744453:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961611008/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079749.744467:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079749.744471:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079749.744475:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961611008:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079749.744481:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961611136:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079749.744500:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079749.744510:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079749/real 1769079749] req@ffff895e4383fb80 x1855008961611136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079749.744521:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961611136/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079749.744528:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079749.744531:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079749.744534:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961611136:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079749.936047:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079750.319913:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079750.960279:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079751.343866:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079751.990820:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079752.367945:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079753.008244:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079753.391853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079754.032369:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079754.415880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079754.864778:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079754.864788:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079754.864792:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079754.864817:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5437 00000100:00080000:1.0:1769079754.864828:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079754.864836:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079754.864872:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961611264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079754.864904:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079754.864907:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079754.864923:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079754.864928:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5437 00000100:00080000:1.0:1769079754.864932:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079754.864936:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079754.864945:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079754.864958:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961611392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079754.864967:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079754.864970:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079754.864972:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079754.864984:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5437 00000100:00080000:1.0:1769079754.864987:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079754.864991:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079754.864995:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079754.865005:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961611520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079754.865160:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961611264:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079754.865242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079754.865284:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079754/real 1769079754] req@ffff895e4713f100 x1855008961611264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079754.865301:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961611264/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079754.865311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079754.865315:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079754.865326:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961611264:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079754.865333:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961611392:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079754.865372:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079754.865385:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079754/real 1769079754] req@ffff895e4713e300 x1855008961611392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079754.865395:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961611392/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079754.865403:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079754.865406:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079754.865409:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961611392:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079754.865415:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961611520:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079754.865433:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079754.865443:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079754/real 1769079754] req@ffff895e4713d880 x1855008961611520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079754.865454:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961611520/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079754.865460:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079754.865463:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079754.865466:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961611520:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079755.056304:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079755.439893:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079756.080180:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079756.464161:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079757.105451:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079757.489272:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079758.127960:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079758.512072:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079759.152335:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079759.536461:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079759.984388:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079759.984399:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079759.984404:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079759.984415:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5442 00000100:00080000:2.0:1769079759.984426:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079759.984433:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079759.984521:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961611648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079759.984556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079759.984559:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079759.984561:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079759.984565:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5442 00000100:00080000:2.0:1769079759.984569:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079759.984573:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079759.984582:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079759.984593:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961611776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079759.984603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079759.984605:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079759.984607:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079759.984611:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5442 00000100:00080000:2.0:1769079759.984614:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079759.984618:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079759.984622:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079759.984649:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961611904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769079759.984733:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961611648:192.168.204.117@tcp:250:kworker.0 00000800:00000100:0.0:1769079759.984833:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079759.984899:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079759/real 1769079759] req@ffff895e44e2bb80 x1855008961611648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079759.984916:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961611648/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079759.984925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079759.984929:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079759.984933:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961611648:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769079759.984940:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961611776:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079759.984961:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079759.984973:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079759/real 1769079759] req@ffff895e44e29c00 x1855008961611776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079759.984982:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961611776/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079759.984990:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079759.984993:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079759.984996:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961611776:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769079759.985001:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961611904:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769079759.985030:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769079759.985040:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079759/real 1769079759] req@ffff895e44e2b800 x1855008961611904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769079759.985063:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961611904/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769079759.985069:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769079759.985072:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769079759.985075:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961611904:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079760.176085:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079760.560539:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079761.203489:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079761.584077:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079762.225080:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079762.607941:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079763.248246:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079763.631876:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079764.272330:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079764.656463:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079765.104026:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079765.104037:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079765.104041:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079765.104051:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5447 00000100:00080000:0.0:1769079765.104061:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079765.104068:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079765.104104:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961612032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079765.104146:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079765.104149:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079765.104151:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079765.104155:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5447 00000100:00080000:0.0:1769079765.104158:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079765.104162:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079765.104169:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079765.104180:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961612160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079765.104203:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079765.104206:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079765.104208:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079765.104211:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5447 00000100:00080000:0.0:1769079765.104215:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079765.104218:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079765.104222:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079765.104231:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961612288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079765.104462:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961612032:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079765.104523:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079765.104558:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079765/real 1769079765] req@ffff895e4383ce00 x1855008961612032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079765.104575:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961612032/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079765.104585:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079765.104589:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079765.104593:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961612032:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079765.104599:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961612160:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079765.104620:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079765.104631:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079765/real 1769079765] req@ffff895e4383f480 x1855008961612160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079765.104642:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961612160/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079765.104657:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079765.104660:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079765.104663:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961612160:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079765.104676:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961612288:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079765.104694:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079765.104712:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079765/real 1769079765] req@ffff895e4383c000 x1855008961612288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079765.104725:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961612288/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079765.104732:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079765.104735:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079765.104738:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961612288:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079765.297080:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079765.679880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079766.322016:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079766.704207:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079767.344124:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079767.727934:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079768.368380:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079768.751892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079769.392149:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079769.775911:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079770.223906:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079770.223916:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079770.223919:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079770.223930:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5452 00000100:00080000:0.0:1769079770.223940:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079770.223996:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079770.224036:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961612416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079770.224073:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079770.224077:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079770.224079:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079770.224082:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5452 00000100:00080000:0.0:1769079770.224100:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079770.224104:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079770.224113:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079770.224126:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961612544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079770.224135:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079770.224138:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079770.224140:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079770.224143:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5452 00000100:00080000:0.0:1769079770.224146:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079770.224149:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079770.224153:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079770.224162:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961612672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079770.224332:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961612416:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079770.224391:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079770.224428:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079770/real 1769079770] req@ffff895e4383dc00 x1855008961612416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079770.224445:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961612416/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079770.224454:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079770.224458:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079770.224462:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961612416:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079770.224476:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961612544:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079770.224502:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079770.224515:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079770/real 1769079770] req@ffff895e4383ca80 x1855008961612544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079770.224527:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961612544/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079770.224534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079770.224538:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079770.224541:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961612544:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079770.224546:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961612672:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079770.224563:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079770.224572:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079770/real 1769079770] req@ffff895e4383ea00 x1855008961612672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079770.224581:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961612672/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079770.224587:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079770.224590:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079770.224593:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961612672:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079770.416094:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079770.800019:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079771.440244:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079771.827352:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079772.464662:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079772.847964:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079773.488311:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079773.871883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079774.513737:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079774.896046:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079775.343898:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079775.343911:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079775.343916:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079775.343927:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5457 00000100:00080000:0.0:1769079775.343939:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079775.343946:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079775.343982:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961612800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079775.344034:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079775.344038:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079775.344041:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079775.344045:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5457 00000100:00080000:0.0:1769079775.344049:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079775.344053:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079775.344062:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079775.344075:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961612928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079775.344084:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079775.344087:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079775.344088:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079775.344091:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5457 00000100:00080000:0.0:1769079775.344094:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079775.344098:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079775.344101:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079775.344110:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961613056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079775.344271:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961612800:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079775.344334:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079775.344373:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079775/real 1769079775] req@ffff895e4383d500 x1855008961612800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079775.344391:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961612800/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079775.344400:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079775.344404:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079775.344408:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961612800:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079775.344415:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961612928:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079775.344435:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079775.344448:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079775/real 1769079775] req@ffff895e4383f800 x1855008961612928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079775.344459:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961612928/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079775.344467:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079775.344470:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079775.344473:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961612928:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079775.344478:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961613056:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079775.344494:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079775.344503:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079775/real 1769079775] req@ffff895e4383e300 x1855008961613056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079775.344524:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961613056/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079775.344530:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079775.344533:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079775.344536:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961613056:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079775.539211:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079775.919886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079776.559962:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079776.943854:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079777.588138:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079777.967941:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079778.611591:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079778.991847:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079779.635064:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079780.015881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:3.0:1769079780.465844:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079780.465855:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769079780.465859:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079780.465869:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5463 00000100:00080000:3.0:1769079780.465891:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079780.465898:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769079780.465932:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961613184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079780.465964:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079780.465967:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769079780.465969:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079780.465973:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5463 00000100:00080000:3.0:1769079780.465977:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079780.466003:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079780.466011:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079780.466023:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7b80 x1855008961613312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769079780.466033:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769079780.466036:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769079780.466038:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769079780.466041:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5463 00000100:00080000:3.0:1769079780.466045:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769079780.466048:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769079780.466052:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769079780.466060:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7100 x1855008961613440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079780.466162:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961613184:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079780.466233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079780.466265:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079780/real 1769079780] req@ffff895e453d5500 x1855008961613184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079780.466281:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961613184/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079780.466289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079780.466292:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079780.466296:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961613184:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079780.466309:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961613312:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079780.466327:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079780.466337:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079780/real 1769079780] req@ffff895e453d7b80 x1855008961613312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079780.466348:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7b80 x1855008961613312/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079780.466361:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079780.466365:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079780.466368:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961613312:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079780.466373:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961613440:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079780.466390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079780.466399:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079780/real 1769079780] req@ffff895e453d7100 x1855008961613440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079780.466410:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7100 x1855008961613440/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079780.466416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079780.466419:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079780.466422:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961613440:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079780.656190:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079781.040379:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079781.680123:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079782.065140:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079782.704069:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079783.087965:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079783.729170:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079784.112115:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079784.752140:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079785.135898:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079785.584843:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079785.584854:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079785.584859:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079785.584869:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5468 00000100:00080000:1.0:1769079785.584880:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079785.584887:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079785.584920:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961613568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079785.584964:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079785.584968:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079785.584970:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079785.584974:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5468 00000100:00080000:1.0:1769079785.584977:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079785.584981:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079785.584990:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079785.585002:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961613696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079785.585011:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079785.585014:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079785.585016:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079785.585019:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5468 00000100:00080000:1.0:1769079785.585022:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079785.585026:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079785.585155:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961613568:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079785.585214:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079785.585258:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079785/real 1769079785] req@ffff895e4713ed80 x1855008961613568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079785.585275:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961613568/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079785.585284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079785.585287:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079785.585291:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961613568:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079785.585297:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961613696:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079785.585319:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079785.585331:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079785/real 1769079785] req@ffff895e4713c000 x1855008961613696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079785.585342:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961613696/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079785.585349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079785.585353:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079785.585356:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961613696:192.168.204.117@tcp:38:kworker.0 00000008:00100000:1.0:1769079785.585422:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079785.585435:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961613824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079785.585503:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961613824:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079785.585527:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079785.585538:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079785/real 1769079785] req@ffff895e4713fb80 x1855008961613824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079785.585549:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961613824/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079785.585569:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079785.585574:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079785.585577:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961613824:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079785.776004:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079786.159970:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079786.801371:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079787.184030:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079787.824667:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079788.207871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079788.848627:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079789.231879:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079789.872472:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079790.256650:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079790.705583:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079790.705592:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079790.705596:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079790.705607:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5473 00000100:00080000:1.0:1769079790.705622:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079790.705631:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079790.705685:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c700 x1855008961613952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079790.705725:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079790.705729:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079790.705730:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079790.705734:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5473 00000100:00080000:1.0:1769079790.705738:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079790.705742:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079790.705768:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079790.705783:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713fb80 x1855008961614080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079790.705793:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1769079790.705806:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961613952:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769079790.705811:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079790.705813:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079790.705816:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5473 00000100:00080000:1.0:1769079790.705820:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079790.705824:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079790.705829:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079790.705838:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961614208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1769079790.705868:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079790.705917:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079790/real 1769079790] req@ffff895e4713c700 x1855008961613952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079790.705933:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c700 x1855008961613952/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079790.705942:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079790.705946:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079790.705949:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961613952:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079790.705956:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961614080:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079790.705977:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079790.705988:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079790/real 1769079790] req@ffff895e4713fb80 x1855008961614080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079790.705999:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713fb80 x1855008961614080/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079790.706006:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079790.706015:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079790.706029:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961614080:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079790.706064:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961614208:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079790.706086:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079790.706097:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079790/real 1769079790] req@ffff895e4713c000 x1855008961614208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079790.706108:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961614208/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079790.706115:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079790.706117:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079790.706120:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961614208:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079790.896233:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079791.282593:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079791.920773:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079792.305234:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079792.943949:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079793.327869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079793.968401:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079794.351868:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079794.992044:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079795.375870:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079795.823909:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079795.823920:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079795.823924:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079795.823934:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5478 00000100:00080000:2.0:1769079795.823944:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079795.823951:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079795.824000:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b800 x1855008961614336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079795.824034:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079795.824037:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079795.824054:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079795.824059:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5478 00000100:00080000:2.0:1769079795.824063:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079795.824067:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079795.824077:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079795.824090:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29c00 x1855008961614464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079795.824099:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079795.824102:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079795.824104:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079795.824107:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5478 00000100:00080000:2.0:1769079795.824111:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079795.824114:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079795.824119:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079795.824128:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2bb80 x1855008961614592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079795.824272:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961614336:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079795.824363:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961614464:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079795.824389:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961614592:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079796.016014:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079796.400013:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079797.039944:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079797.423878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079798.063963:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079798.447943:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769079798.901113:0:2392:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769079798.901162:0:2392:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769079798.901181:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769079798.905809:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079795/real 1769079798] req@ffff895e44e2b800 x1855008961614336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079820 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079798.905845:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b800 x1855008961614336/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079820 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079798.905856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079798.905861:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079798.905865:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961614336:192.168.204.117@tcp:250:kworker.0 00000800:00000100:1.0:1769079798.906910:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079798.906933:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769079798.907510:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079795/real 1769079798] req@ffff895e44e29c00 x1855008961614464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079798.907533:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29c00 x1855008961614464/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079798.907543:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079798.907547:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079798.907550:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961614464:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079798.907557:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079795/real 1769079798] req@ffff895e44e2bb80 x1855008961614592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079798.907568:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2bb80 x1855008961614592/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079798.907575:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079798.907578:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079798.907581:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961614592:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079799.088679:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079799.471864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079800.114002:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079800.497207:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079800.943989:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079800.944001:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079800.944005:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079800.944034:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5483 00000100:00080000:0.0:1769079800.944046:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079800.944054:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079800.944092:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e300 x1855008961614720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079800.944125:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079800.944129:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079800.944131:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079800.944135:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5483 00000100:00080000:0.0:1769079800.944139:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079800.944142:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079800.944151:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079800.944164:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f800 x1855008961614848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079800.944173:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079800.944177:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079800.944178:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079800.944182:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5483 00000100:00080000:0.0:1769079800.944185:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079800.944188:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079800.944192:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079800.944202:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d500 x1855008961614976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079800.944320:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961614720:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079800.944408:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079800.944444:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079800/real 1769079800] req@ffff895e4383e300 x1855008961614720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079800.944463:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e300 x1855008961614720/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079800.944473:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079800.944477:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079800.944481:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961614720:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079800.944501:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961614848:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079800.944524:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079800.944536:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079800/real 1769079800] req@ffff895e4383f800 x1855008961614848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079800.944548:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f800 x1855008961614848/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079800.944556:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079800.944559:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079800.944562:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961614848:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079800.944568:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961614976:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079800.944585:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079800.944595:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079800/real 1769079800] req@ffff895e4383d500 x1855008961614976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079800.944613:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d500 x1855008961614976/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079800.944621:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079800.944623:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079800.944627:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961614976:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769079801.136148:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079801.520251:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079802.159975:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079802.544020:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079803.184135:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079803.568086:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:2.0:1769079804.208284:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079804.592596:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079805.233880:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079805.615957:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079806.064115:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079806.064124:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079806.064128:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079806.064138:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5488 00000100:00080000:2.0:1769079806.064148:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079806.064154:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079806.064205:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29500 x1855008961615104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079806.064244:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079806.064248:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079806.064250:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079806.064253:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5488 00000100:00080000:2.0:1769079806.064259:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079806.064262:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079806.064272:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079806.064283:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28a80 x1855008961615232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079806.064293:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961615104:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769079806.064295:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079806.064303:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079806.064305:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079806.064309:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5488 00000100:00080000:2.0:1769079806.064312:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079806.064316:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079806.064321:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079806.064333:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29180 x1855008961615360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769079806.064364:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079806.064504:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079806/real 1769079806] req@ffff895e44e29500 x1855008961615104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079806.064522:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29500 x1855008961615104/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079806.064532:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079806.064537:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079806.064541:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961615104:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079806.064574:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961615232:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079806.064596:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079806.064608:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079806/real 1769079806] req@ffff895e44e28a80 x1855008961615232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079806.064620:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28a80 x1855008961615232/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079806.064640:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079806.064643:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079806.064646:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961615232:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079806.064652:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961615360:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079806.064670:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079806.064680:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079806/real 1769079806] req@ffff895e44e29180 x1855008961615360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079806.064689:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29180 x1855008961615360/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079806.064695:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079806.065544:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079806.065551:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961615360:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079806.257182:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079806.640516:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079807.283603:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079807.665903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079808.305880:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079808.690384:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079809.328089:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079809.711886:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079810.353726:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079810.735994:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079811.184036:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079811.184047:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079811.184053:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079811.184064:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5493 00000100:00080000:1.0:1769079811.184077:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079811.184084:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079811.184135:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961615488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079811.184173:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079811.184177:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079811.184197:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079811.184201:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5493 00000100:00080000:1.0:1769079811.184205:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079811.184209:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079811.184219:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079811.184232:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961615616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079811.184241:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079811.184244:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079811.184245:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079811.184248:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5493 00000100:00080000:1.0:1769079811.184251:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079811.184254:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079811.184258:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079811.184266:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961615744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079811.184360:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961615488:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079811.184427:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079811.184468:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079811/real 1769079811] req@ffff895e4713ed80 x1855008961615488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079811.184498:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961615488/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079811.184509:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079811.184513:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079811.184524:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961615488:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079811.184531:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961615616:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079811.184554:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079811.184566:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079811/real 1769079811] req@ffff895e4713d880 x1855008961615616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079811.184577:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961615616/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079811.184584:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079811.184587:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079811.184590:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961615616:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079811.184596:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961615744:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079811.184613:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079811.184622:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079811/real 1769079811] req@ffff895e4713e300 x1855008961615744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079811.184631:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961615744/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079811.184638:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079811.184640:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079811.184643:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961615744:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079811.376780:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079811.759913:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079812.407369:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079812.784427:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079813.424243:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079813.808483:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079814.448693:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079814.832012:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079815.472524:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079815.855858:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079816.304157:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079816.304166:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079816.304171:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079816.304199:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5498 00000100:00080000:2.0:1769079816.304211:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079816.304234:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079816.304277:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29f80 x1855008961615872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079816.304306:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079816.304309:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079816.304311:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079816.304315:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5498 00000100:00080000:2.0:1769079816.304319:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079816.304324:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079816.304332:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079816.304343:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28e00 x1855008961616000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079816.304353:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079816.304356:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079816.304357:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079816.304360:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5498 00000100:00080000:2.0:1769079816.304364:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079816.304367:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079816.304371:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079816.304379:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b480 x1855008961616128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079816.305168:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961615872:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079816.305248:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079816.305290:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079816/real 1769079816] req@ffff895e44e29f80 x1855008961615872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079816.305306:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29f80 x1855008961615872/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079816.305315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079816.305319:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079816.305323:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961615872:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079816.305329:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961616000:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079816.305349:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079816.305361:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079816/real 1769079816] req@ffff895e44e28e00 x1855008961616000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079816.305372:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28e00 x1855008961616000/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079816.305380:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079816.305382:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079816.305386:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961616000:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079816.305391:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961616128:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079816.305409:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079816.305420:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079816/real 1769079816] req@ffff895e44e2b480 x1855008961616128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079816.305441:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b480 x1855008961616128/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079816.305447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079816.305450:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079816.305453:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961616128:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079816.495927:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079816.880156:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079817.520375:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079817.905111:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079818.544278:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079818.927878:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079819.570262:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079819.951904:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079820.592393:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079820.977156:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079821.423992:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079821.424027:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079821.424031:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079821.424041:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5503 00000100:00080000:1.0:1769079821.424052:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079821.424058:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079821.424092:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961616256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079821.424169:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961616256:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079821.424225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00080000:1.0:1769079821.424256:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079821.424259:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079821.424261:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079821.424265:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5503 00000100:00080000:1.0:1769079821.424269:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079821.424273:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079821.424281:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079821.424293:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961616384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079821.429690:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079821/real 1769079821] req@ffff895e4713f100 x1855008961616256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079821.429719:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961616256/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079821.429730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079821.429734:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079821.429738:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961616256:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079821.429795:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961616384:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079821.430456:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079821.430476:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079821/real 1769079821] req@ffff895e4713ea00 x1855008961616384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079821.430587:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961616384/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079821.430596:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079821.430599:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079821.430602:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961616384:192.168.204.117@tcp:38:kworker.0 00000100:00080000:1.0:1769079821.433391:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079821.433402:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079821.433405:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079821.433414:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5503 00000100:00080000:1.0:1769079821.433425:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079821.433432:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079821.433443:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079821.433477:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961616512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079821.435557:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961616512:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079821.435630:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079821.435660:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079821/real 1769079821] req@ffff895e4713ce00 x1855008961616512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079821.435690:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961616512/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079821.435701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079821.435705:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079821.435709:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961616512:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079821.616781:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079822.000613:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079822.642475:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079823.024230:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079823.664572:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079824.048165:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079824.689155:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079825.072363:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079825.715148:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079826.095975:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079826.545239:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079826.545249:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079826.545253:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079826.545262:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5509 00000100:00080000:1.0:1769079826.545273:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079826.545279:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079826.545314:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961616640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079826.545354:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079826.545357:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079826.545359:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079826.545362:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5509 00000100:00080000:1.0:1769079826.545365:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079826.545368:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079826.545377:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079826.545389:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961616768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079826.545398:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079826.545404:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1769079826.545405:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961616640:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769079826.545406:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1769079826.545461:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079826.545498:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079826/real 1769079826] req@ffff895e4713d500 x1855008961616640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079826.545524:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961616640/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079826.545533:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079826.545537:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079826.545541:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961616640:192.168.204.117@tcp:250:kworker.0 00000100:00080000:1.0:1769079826.545561:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5509 00000100:00080000:1.0:1769079826.545567:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079826.545571:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079826.545576:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079826.545588:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961616896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079826.545620:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961616768:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079826.545642:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079826.545653:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079826/real 1769079826] req@ffff895e4713f480 x1855008961616768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079826.545663:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961616768/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079826.545671:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079826.545674:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079826.545676:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961616768:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079826.545681:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961616896:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079826.545695:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079826.545703:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079826/real 1769079826] req@ffff895e4713ca80 x1855008961616896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079826.545712:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961616896/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079826.545728:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079826.545731:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079826.545733:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961616896:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079826.738922:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079827.120108:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079827.760075:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079828.144176:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079828.783990:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079829.168981:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079829.808484:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079830.192540:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079830.838916:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079831.215888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079831.664021:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079831.664031:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079831.664036:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079831.664046:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5514 00000100:00080000:2.0:1769079831.664068:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079831.664077:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079831.664116:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2b100 x1855008961617024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079831.664148:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079831.664151:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079831.664153:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079831.664157:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5514 00000100:00080000:2.0:1769079831.664162:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079831.664165:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079831.664174:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079831.664186:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2a300 x1855008961617152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079831.664195:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079831.664198:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079831.664200:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1769079831.664201:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961617024:192.168.204.117@tcp:250:kworker.0 00000100:00080000:2.0:1769079831.664204:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5514 00000100:00080000:2.0:1769079831.664207:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079831.664210:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079831.664229:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079831.664239:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e2aa00 x1855008961617280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769079831.664272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079831.664306:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079831/real 1769079831] req@ffff895e44e2b100 x1855008961617024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079831.664323:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2b100 x1855008961617024/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079831.664332:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079831.664336:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079831.664340:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961617024:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079831.664364:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961617152:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079831.664386:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079831.664398:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079831/real 1769079831] req@ffff895e44e2a300 x1855008961617152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079831.664410:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2a300 x1855008961617152/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079831.664425:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079831.664429:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079831.664432:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961617152:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079831.664438:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961617280:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079831.664458:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079831.664469:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079831/real 1769079831] req@ffff895e44e2aa00 x1855008961617280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079831.664480:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e2aa00 x1855008961617280/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079831.664488:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079831.664491:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079831.664494:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e2aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961617280:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079831.856454:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079832.239906:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079832.880211:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079833.264694:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079833.904044:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079834.290344:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079834.928043:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079835.311885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079835.953888:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079836.337162:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079836.784294:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079836.784305:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079836.784310:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079836.784320:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5519 00000100:00080000:1.0:1769079836.784331:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079836.784337:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079836.784375:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961617408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079836.784413:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079836.784416:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079836.784419:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079836.784423:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5519 00000100:00080000:1.0:1769079836.784443:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079836.784448:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079836.784457:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079836.784470:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961617536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079836.784480:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079836.784483:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079836.784485:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079836.784488:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5519 00000100:00080000:1.0:1769079836.784491:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079836.784496:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079836.784500:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079836.784509:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961617664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079836.784590:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961617408:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079836.784650:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079836.784711:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079836/real 1769079836] req@ffff895e4713f800 x1855008961617408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079836.784737:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961617408/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079836.784786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079836.784790:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079836.784794:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961617408:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079836.784801:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961617536:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079836.784824:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079836.784836:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079836/real 1769079836] req@ffff895e4713dc00 x1855008961617536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079836.784848:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961617536/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079836.784855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079836.784859:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079836.784862:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961617536:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079836.784867:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961617664:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079836.784884:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079836.784893:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079836/real 1769079836] req@ffff895e4713df80 x1855008961617664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079836.784904:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961617664/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079836.784911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079836.784914:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079836.784917:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961617664:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079836.976022:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079837.360436:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079837.999936:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079838.383892:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079839.023932:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079839.407865:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079840.047963:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079840.431885:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079841.072113:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079841.455932:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079841.903863:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079841.903873:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079841.903878:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079841.903888:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5524 00000100:00080000:1.0:1769079841.903898:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079841.903905:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079841.903942:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961617792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079841.903988:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079841.903992:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079841.903994:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079841.903998:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5524 00000100:00080000:1.0:1769079841.904002:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079841.904007:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079841.904016:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079841.904028:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961617920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079841.904038:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079841.904041:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079841.904043:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079841.904046:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5524 00000100:00080000:1.0:1769079841.904050:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079841.904053:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079841.904058:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079841.904068:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961618048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079841.904156:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961617792:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079841.904214:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079841.904249:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079841/real 1769079841] req@ffff895e4713d180 x1855008961617792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079841.904265:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961617792/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079841.904273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079841.904277:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079841.904281:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961617792:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079841.904288:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961617920:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079841.904308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079841.904319:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079841/real 1769079841] req@ffff895e4713c380 x1855008961617920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079841.904330:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961617920/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079841.904337:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079841.904340:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079841.904343:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961617920:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079841.904358:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961618048:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079841.904378:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079841.904388:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079841/real 1769079841] req@ffff895e4713df80 x1855008961618048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079841.904399:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961618048/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079841.904407:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079841.904410:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079841.904413:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961618048:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079842.097296:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079842.479873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079843.120282:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079843.504408:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079844.144393:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079844.527905:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079845.167975:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079845.551846:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079846.197360:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079846.576051:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079847.025175:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079847.025187:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079847.025192:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079847.025202:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5529 00000100:00080000:0.0:1769079847.025215:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079847.025222:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079847.025264:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961618176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079847.025300:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079847.025304:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079847.025306:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079847.025310:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5529 00000100:00080000:0.0:1769079847.025314:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079847.025332:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079847.025342:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079847.025356:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961618304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079847.025366:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079847.025370:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079847.025372:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079847.025375:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5529 00000100:00080000:0.0:1769079847.025379:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079847.025384:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079847.025388:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079847.025397:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961618432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079847.025483:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961618176:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079847.025542:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079847.025593:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079847/real 1769079847] req@ffff895e4383ea00 x1855008961618176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079847.025610:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961618176/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079847.025619:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079847.025622:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079847.025626:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961618176:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079847.025639:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961618304:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079847.025660:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079847.025671:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079847/real 1769079847] req@ffff895e4383ca80 x1855008961618304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079847.025683:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961618304/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079847.025691:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079847.025694:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079847.025697:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961618304:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079847.025703:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961618432:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079847.025719:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079847.025729:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079847/real 1769079847] req@ffff895e4383dc00 x1855008961618432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079847.025741:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961618432/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079847.025773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079847.025776:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079847.025779:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961618432:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079847.216525:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079847.599872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079848.240443:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079848.627780:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079849.264123:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079849.647910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079850.288222:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079850.671995:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079851.312102:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079851.696084:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079852.144053:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079852.144065:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079852.144069:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079852.144079:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5534 00000100:00080000:2.0:1769079852.144093:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079852.144099:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079852.144135:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961618560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079852.144167:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079852.144170:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079852.144172:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079852.144176:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5534 00000100:00080000:2.0:1769079852.144179:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079852.144183:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079852.144192:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079852.144202:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961618688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079852.144211:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079852.144214:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079852.144216:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079852.144219:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5534 00000100:00080000:2.0:1769079852.144239:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079852.144244:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079852.144249:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079852.144259:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961618816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079852.144408:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961618560:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079852.144474:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079852.144508:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079852/real 1769079852] req@ffff895e44e28000 x1855008961618560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079852.144524:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961618560/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079852.144532:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079852.144536:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079852.144539:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961618560:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079852.144546:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961618688:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079852.144565:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079852.144575:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079852/real 1769079852] req@ffff895e44e28380 x1855008961618688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079852.144584:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961618688/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079852.144591:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079852.144594:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079852.144596:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961618688:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079852.144601:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961618816:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079852.144614:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079852.144623:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079852/real 1769079852] req@ffff895e44e29880 x1855008961618816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079852.144640:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961618816/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079852.144658:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079852.144661:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079852.144664:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961618816:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079852.336817:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079852.721853:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079853.359954:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079853.743894:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079854.385136:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079854.769000:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079855.409419:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079855.792849:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079856.432133:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079856.815950:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079857.265008:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079857.265019:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079857.265024:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079857.265033:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5539 00000100:00080000:1.0:1769079857.265045:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079857.265052:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079857.265088:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961618944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079857.265122:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079857.265126:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079857.265128:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079857.265132:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5539 00000100:00080000:1.0:1769079857.265135:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079857.265139:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079857.265147:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079857.265159:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961619072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079857.265168:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079857.265172:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079857.265173:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079857.265177:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5539 00000100:00080000:1.0:1769079857.265180:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079857.265198:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079857.265203:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079857.265213:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961619200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079857.266456:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961618944:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079857.266523:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079857.266561:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079857/real 1769079857] req@ffff895e4713dc00 x1855008961618944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079857.266579:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961618944/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079857.266590:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079857.266594:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079857.266598:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961618944:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079857.266605:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961619072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079857.266627:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079857.266639:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079857/real 1769079857] req@ffff895e4713f800 x1855008961619072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079857.266664:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961619072/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079857.266672:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079857.266676:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079857.266679:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961619072:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079857.266685:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961619200:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079857.266705:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079857.266715:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079857/real 1769079857] req@ffff895e4713ca80 x1855008961619200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079857.266726:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961619200/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079857.266733:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079857.266736:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079857.266739:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961619200:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079857.456496:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079857.839873:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079858.485816:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079858.865066:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079859.505039:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079859.889078:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079860.528871:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079860.911907:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079861.551919:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079861.935888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079862.383873:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079862.383884:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079862.383888:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079862.383898:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5544 00000100:00080000:1.0:1769079862.383909:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079862.383917:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079862.383958:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961619328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079862.383995:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079862.383999:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079862.384001:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079862.384005:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5544 00000100:00080000:1.0:1769079862.384008:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079862.384012:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079862.384021:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079862.384032:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961619456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079862.384056:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079862.384059:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079862.384061:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079862.384065:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5544 00000100:00080000:1.0:1769079862.384069:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079862.384073:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079862.384079:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079862.384091:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961619584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079862.384190:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961619328:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079862.384300:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961619456:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079862.384327:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961619584:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079862.576325:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1769079862.959881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079863.600280:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079863.983979:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079864.623955:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079865.011229:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000400:00000100:1.0:1769079865.456485:0:2390:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769079865.456527:0:2390:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769079865.456543:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079865.456842:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769079865.456852:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:2.0:1769079865.457289:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079862/real 1769079865] req@ffff895e4713f480 x1855008961619328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079865.457312:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961619328/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079865.457323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079865.457327:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079865.457331:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961619328:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079865.457339:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079862/real 1769079865] req@ffff895e4713d500 x1855008961619456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079865.457349:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961619456/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079865.457357:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079865.457360:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079865.457363:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961619456:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079865.457369:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079862/real 1769079865] req@ffff895e4713ce00 x1855008961619584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079865.457379:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961619584/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079865.457401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079865.457404:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079865.457407:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961619584:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079865.648098:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079866.031887:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079866.673395:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079867.055880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079867.507339:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079867.507349:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079867.507353:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079867.507361:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5550 00000100:00080000:0.0:1769079867.507373:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079867.507380:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079867.507414:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c000 x1855008961619712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079867.507449:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079867.507453:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079867.507455:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079867.507459:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5550 00000100:00080000:0.0:1769079867.507462:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079867.507466:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079867.507479:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079867.507491:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f480 x1855008961619840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079867.507500:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079867.507503:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079867.507505:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079867.507508:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5550 00000100:00080000:0.0:1769079867.507511:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079867.507514:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079867.507519:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079867.507546:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ce00 x1855008961619968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1769079867.507771:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961619712:192.168.204.117@tcp:250:kworker.0 00000800:00000100:2.0:1769079867.507826:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079867.507856:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079867/real 1769079867] req@ffff895e4383c000 x1855008961619712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079867.507872:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c000 x1855008961619712/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079867.507881:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079867.507885:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079867.507889:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961619712:192.168.204.117@tcp:250:kworker.0 00000100:00100000:2.0:1769079867.507895:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961619840:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079867.507910:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079867.507919:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079867/real 1769079867] req@ffff895e4383f480 x1855008961619840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079867.507929:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f480 x1855008961619840/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079867.507937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079867.507939:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079867.507942:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961619840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:2.0:1769079867.507953:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961619968:192.168.204.117@tcp:38:kworker.0 00000800:00000100:2.0:1769079867.507968:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:2.0:1769079867.507976:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079867/real 1769079867] req@ffff895e4383ce00 x1855008961619968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1769079867.507986:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ce00 x1855008961619968/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1769079867.507994:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1769079867.507996:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:2.0:1769079867.507999:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961619968:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079867.696060:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079868.079871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079868.719980:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079869.104781:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079869.745488:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079870.129245:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079870.768049:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079871.152884:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079871.791940:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079872.175877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:2.0:1769079872.624199:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079872.624211:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1769079872.624216:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079872.624225:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5555 00000100:00080000:2.0:1769079872.624237:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079872.624244:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:2.0:1769079872.624284:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e29880 x1855008961620096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079872.624320:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079872.624324:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1769079872.624326:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079872.624330:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5555 00000100:00080000:2.0:1769079872.624334:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079872.624353:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079872.624362:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079872.624374:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28380 x1855008961620224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1769079872.624384:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1769079872.624387:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1769079872.624389:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1769079872.624392:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5555 00000100:00080000:2.0:1769079872.624396:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1769079872.624399:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:2.0:1769079872.624403:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1769079872.624412:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e44e28000 x1855008961620352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079872.624637:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961620096:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079872.624788:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079872.624882:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079872/real 1769079872] req@ffff895e44e29880 x1855008961620096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079872.624914:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e29880 x1855008961620096/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079872.624924:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079872.624941:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079872.624945:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e29880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961620096:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079872.624952:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961620224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079872.624998:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079872.625012:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079872/real 1769079872] req@ffff895e44e28380 x1855008961620224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079872.625039:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28380 x1855008961620224/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079872.625047:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079872.625060:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079872.625071:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961620224:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079872.625093:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961620352:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079872.625114:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079872.625152:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079872/real 1769079872] req@ffff895e44e28000 x1855008961620352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079872.625163:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e44e28000 x1855008961620352/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079872.625171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079872.625174:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079872.625178:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e44e28000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961620352:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079872.816005:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079873.199877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079873.840117:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079874.223872:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079874.864062:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079875.251725:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079875.889090:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079876.271864:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079876.912277:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079877.295856:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079877.745623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079877.745633:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079877.745638:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079877.745647:0:37:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5560 00000100:00080000:0.0:1769079877.745659:0:37:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079877.745665:0:37:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079877.745702:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383fb80 x1855008961620480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079877.745732:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079877.745735:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079877.745737:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079877.745741:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5560 00000100:00080000:0.0:1769079877.745768:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079877.745773:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079877.745782:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079877.745794:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c380 x1855008961620608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079877.745823:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079877.745826:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079877.745828:0:37:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079877.745831:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5560 00000100:00080000:0.0:1769079877.745834:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079877.745838:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079877.745842:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079877.745852:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383f100 x1855008961620736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079877.746019:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961620480:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079877.746091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079877.746129:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079877/real 1769079877] req@ffff895e4383fb80 x1855008961620480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079877.746146:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383fb80 x1855008961620480/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079877.746156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079877.746160:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079877.746171:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961620480:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079877.746178:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961620608:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079877.746201:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079877.746212:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079877/real 1769079877] req@ffff895e4383c380 x1855008961620608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079877.746223:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c380 x1855008961620608/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079877.746231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079877.746234:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079877.746237:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961620608:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079877.746243:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961620736:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079877.746258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079877.746268:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079877/real 1769079877] req@ffff895e4383f100 x1855008961620736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079877.746278:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383f100 x1855008961620736/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079877.746285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079877.746288:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079877.746292:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961620736:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079877.942224:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079878.319875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079878.960330:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079879.345071:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079879.983955:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079880.367880:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079881.007966:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079881.391859:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079882.032532:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079882.415877:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079882.863971:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079882.863982:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079882.863988:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079882.863999:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5565 00000100:00080000:1.0:1769079882.864029:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079882.864037:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079882.864068:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961620864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079882.864096:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079882.864100:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079882.864102:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079882.864106:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5565 00000100:00080000:1.0:1769079882.864109:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079882.864113:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079882.864121:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079882.864131:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961620992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079882.864140:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079882.864143:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079882.864144:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079882.864147:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5565 00000100:00080000:1.0:1769079882.864150:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079882.864154:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079882.864159:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079882.864166:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961621120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079882.864246:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961620864:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079882.864307:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079882.864343:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079882/real 1769079882] req@ffff895e4713ea00 x1855008961620864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079882.864360:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961620864/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079882.864369:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079882.864373:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079882.864377:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961620864:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079882.864384:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961620992:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079882.864403:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079882.864414:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079882/real 1769079882] req@ffff895e4713f100 x1855008961620992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079882.864425:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961620992/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079882.864433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079882.864436:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079882.864439:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961620992:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079882.864444:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961621120:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079882.864462:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079882.864480:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079882/real 1769079882] req@ffff895e4713e300 x1855008961621120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079882.864491:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961621120/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079882.864498:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079882.864501:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079882.864504:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961621120:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079883.060326:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079883.439869:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079884.080450:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079884.464013:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079885.104284:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079885.488273:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079886.129161:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079886.511910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079887.158702:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079887.535903:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:1.0:1769079887.984044:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079887.984052:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769079887.984056:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079887.984064:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5570 00000100:00080000:1.0:1769079887.984073:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079887.984078:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769079887.984117:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d880 x1855008961621248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079887.984141:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079887.984144:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769079887.984146:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079887.984150:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5570 00000100:00080000:1.0:1769079887.984153:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079887.984156:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079887.984164:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079887.984176:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ed80 x1855008961621376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769079887.984182:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769079887.984185:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769079887.984186:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769079887.984188:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5570 00000100:00080000:1.0:1769079887.984190:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769079887.984192:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769079887.984210:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769079887.984217:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c000 x1855008961621504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079887.985545:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961621248:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079887.985610:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079887.985642:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079887/real 1769079887] req@ffff895e4713d880 x1855008961621248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079887.985719:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d880 x1855008961621248/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079887.985727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079887.985730:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079887.985733:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961621248:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079887.985738:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961621376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079887.985796:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079887.985944:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079887/real 1769079887] req@ffff895e4713ed80 x1855008961621376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079887.985957:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ed80 x1855008961621376/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079887.985966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079887.986039:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079887.986043:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961621376:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079887.986050:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961621504:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079887.986072:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079887.986083:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079887/real 1769079887] req@ffff895e4713c000 x1855008961621504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079887.986093:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c000 x1855008961621504/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079887.986100:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079887.986103:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079887.986194:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961621504:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079888.176293:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079888.560067:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079889.199964:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079889.584254:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079890.223951:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079890.609402:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079891.249170:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079891.631871:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079892.272900:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079892.655974:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079893.105196:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079893.105206:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079893.105210:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079893.105219:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5575 00000100:00080000:0.0:1769079893.105229:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079893.105236:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079893.105286:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383c700 x1855008961621632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079893.105317:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079893.105321:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079893.105322:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079893.105326:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5575 00000100:00080000:0.0:1769079893.105331:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079893.105335:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079893.105344:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079893.105355:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383e680 x1855008961621760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079893.105364:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079893.105367:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079893.105369:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079893.105372:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5575 00000100:00080000:0.0:1769079893.105375:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079893.105378:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079893.105383:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079893.105393:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d880 x1855008961621888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079893.105483:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961621632:192.168.204.117@tcp:250:kworker.0 00000800:00000100:3.0:1769079893.105543:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079893.105588:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079893/real 1769079893] req@ffff895e4383c700 x1855008961621632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079893.105607:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383c700 x1855008961621632/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079893.105628:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079893.105633:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079893.105637:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961621632:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079893.105645:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961621760:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079893.105667:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079893.105680:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079893/real 1769079893] req@ffff895e4383e680 x1855008961621760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079893.105692:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383e680 x1855008961621760/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079893.105700:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079893.105703:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079893.105707:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961621760:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079893.105712:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961621888:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079893.105731:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079893.105742:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079893/real 1769079893] req@ffff895e4383d880 x1855008961621888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079893.105793:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d880 x1855008961621888/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079893.105812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079893.105815:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079893.105818:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961621888:192.168.204.117@tcp:38:kworker.0 00010000:00010000:0.0:1769079893.296275:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079893.680649:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079894.323537:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079894.703910:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079895.343942:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079895.727875:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079896.367986:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079896.752076:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079897.392160:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079897.776080:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00000100:00080000:0.0:1769079898.224166:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079898.224177:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769079898.224182:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079898.224192:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 5580 00000100:00080000:0.0:1769079898.224204:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079898.224212:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769079898.224249:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ed80 x1855008961622016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769079898.224283:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079898.224286:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769079898.224288:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079898.224293:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5580 00000100:00080000:0.0:1769079898.224297:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079898.224301:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079898.224310:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079898.224324:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383df80 x1855008961622144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769079898.224330:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961622016:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769079898.224337:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769079898.224342:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769079898.224344:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769079898.224348:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 5580 00000100:00080000:0.0:1769079898.224352:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769079898.224356:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769079898.224360:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769079898.224390:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383d180 x1855008961622272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769079898.224400:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079898.224499:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079898/real 1769079898] req@ffff895e4383ed80 x1855008961622016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079898.224517:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ed80 x1855008961622016/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769079923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079898.224527:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079898.224532:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079898.224536:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961622016:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769079898.224559:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961622144:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079898.224582:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079898.224594:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079898/real 1769079898] req@ffff895e4383df80 x1855008961622144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079898.224607:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383df80 x1855008961622144/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079898.224621:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079898.224625:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079898.224628:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961622144:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769079898.224634:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961622272:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769079898.224654:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769079898.224665:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769079898/real 1769079898] req@ffff895e4383d180 x1855008961622272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769079898.224676:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383d180 x1855008961622272/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769079923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769079898.224682:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769079898.224713:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769079898.224717:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961622272:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769079898.417320:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079898.799883:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079899.441154:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079899.823881:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:0.0:1769079900.463950:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079900.848118:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds 00010000:00010000:1.0:1769079901.488140:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1769079901.871888:0:28878:0:(client.c:2690:ptlrpc_set_wait()) set ffff895e452c36c0 going to sleep for 0 seconds Debug log: 38181 lines, 38181 kept, 0 dropped, 0 bad.